DEV Community: DevOps Daily

Hacktoberfest 2026 Stopped Counting PRs. Make Your First Ones Anyway, One a Day

DevOps Daily — Thu, 01 Oct 2026 15:04:37 +0000

Hacktoberfest started today, and if you have taken part before, the first thing to know is that the rule everyone remembers is gone. Pull requests no longer count toward Hacktoberfest rewards. There is no PR target and no PR-based swag this year.

That is a good change. It is also a slightly awkward one if Hacktoberfest was the push you needed to make your first open source contribution. This post covers what changed, why a first PR is still worth making, and a 7-day challenge we built: one small task a day, 5 to 15 minutes each, and each one ends with a real pull request that a maintainer reviews.

What changed in Hacktoberfest 2026

From the official FAQ:

Pull requests and merge requests will no longer count toward Hacktoberfest rewards. It’s easier than ever to submit low-effort spam PRs to projects, so we’re listening to maintainer feedback and no longer actively incentivizing PRs.

The rest of the event changed with it:

New organizers. MLH and DEV run Hacktoberfest this year, in partnership with DigitalOcean.
New format. 300+ in-person and online community events, focused on hands-on building and learning with open-source AI and open-weight models. The in-person ones are called Fests.
New rewards. You collect stickers. Two are required: sign in with MyMLH and add your mailing address. You earn more by checking in at a Fest, entering the code shown during a livestream, or submitting to a DEV Challenge. Fifteen stickers enter you in a raffle for a Hacktoberfest t-shirt or an Arduino Uno Q board (details on the online participation page).

If you maintain a repo, you know why. Every October, maintainers spent hours closing pull requests that changed one word in a README, and AI tools made those PRs even cheaper to produce. Taking the swag off the PR removes the main reason to send them.

Why your first PR still matters

The spam was the problem, not the pull request. The things a first contribution teaches you did not change:

Forking, branching and keeping a fork in sync without breaking it.
Reading a codebase you did not write, well enough to change one thing in it.
Running a project's tests before a reviewer has to tell you they fail.
Writing a PR description that a busy maintainer can approve in one read.
Taking review feedback and pushing a fix to the same branch.

Those are daily skills in any DevOps or platform job. A merged PR in a public repo is also evidence of them, which a line on a CV is not.

What you need is a project that wants small contributions, says exactly how to make them, and reviews them. So we made one.

The challenge: one small PR a day for 7 days

The DevOps Daily Hacktoberfest challenge is 7 daily tasks, plus a bonus, in the devops-daily repo: an open source DevOps learning site with 560+ posts, 45 quizzes and 27 flashcard decks, all stored as Markdown and JSON in the repo. Each day has its own page with the file to change and an example, and most days include a command to check your work.

To be clear about what this is: it is a DevOps Daily challenge that runs during Hacktoberfest, not an official Hacktoberfest event, and these PRs do not count toward Hacktoberfest stickers. What you get is a real contribution, reviewed by a maintainer, with a test suite in CI, on your GitHub profile. The first challenge PRs came in within hours.

Set up once (about 10 minutes)

You need Git, Node.js 22.13.1 or later (below 25), and pnpm 10.

# Fork https://github.com/The-DevOps-Daily/devops-daily on GitHub first, then:
git clone https://github.com/<your-username>/devops-daily.git
cd devops-daily
git remote add upstream https://github.com/The-DevOps-Daily/devops-daily.git

pnpm install
pnpm test          # the unit tests CI runs on every PR

Before each day's task, start a new branch from a fresh copy of main, so your PR contains only that day's change. Change the day number each time:

git fetch upstream
git checkout -b hacktoberfest/day-1 upstream/main

The one exception is Day 7, which adds to your Day 1 profile. If your Day 1 PR is not merged yet, branch from your Day 1 branch instead.

The 7 days

Day 1: Add yourself to the experts directory (5 min). Create a Markdown profile in content/experts/ with your name, a short bio and how people can reach you. It is the smallest possible PR, and it shows you the whole fork, branch, commit, push and PR flow once before the content gets harder.

Day 2: Add your favorite DevOps tool (5 min). Add one entry to the toolbox page: name, one-line description, link and category. Pick a tool you use, not one you have heard of. The description is where your experience shows.

Day 3: Add a quiz question (5 min). Add a question to one of the existing quizzes in content/quizzes/. The tests check every question's shape, so run them before you push. Write the explanation for the wrong answers too: "why not B" teaches more than "why A".

Day 4: Add a flashcard (5 min). Add one or two cards to a deck in content/flashcards/. A good card has one fact on the front and the shortest complete answer on the back.

Day 5: Share a tip (10 min). Add a gotcha or "thing I wish I knew" to an existing post or guide. This is the first day where you edit someone else's writing, so add your tip where a reader needs it and leave the rest of the post alone.

Day 6: Find and fix something (10 min). Browse devops-daily.com and find a typo, a broken link, a deprecated command or a formatting problem. Fix exactly one thing, and put the URL where you found it in the PR description.

Day 7: Share your stack (15 min). Add a "My Stack" section to the profile you made on Day 1: what you run, why you picked it, and what you would change.

Bonus: Build something (30+ min). A full quiz on a topic that is not covered yet, a tool comparison, a production checklist, or for the ambitious, an interactive game or simulator.

How to get your PR merged (and not closed as spam)

The 2026 change is a good reminder of what maintainers are tired of. These habits make a PR easy to review in any project, not only this one:

One change per PR. If you spot a typo in another quiz while doing Day 3, fix it in a separate PR. A small diff is quick to check.
Run the tests first. Run pnpm test before you push, so you find a failing check before a reviewer does.
Say what and why. Two sentences: what you changed, and why it is correct. For a fix, add the link to where you found the problem.
Read what you submit. If an AI tool helped you write a quiz question or a tip, check it like you would check a stranger's code. A confident wrong answer in a quiz is worse than no question.
Answer review on the same branch. Push the fix to the same branch and the PR updates. Do not open a new PR for each round.

Collect your Hacktoberfest stickers too

The challenge and the official event go together well. Do one small PR a day here, and collect your stickers at hacktoberfest.com: check in at a Fest near you, enter the code during a livestream, or submit to a DEV Challenge, since DEV co-runs the event this year.

Start with Day 1 today: devops-daily.com/hacktoberfest. And if you get stuck on any day, open an issue on the repo or ask in your PR. Questions are contributions too.

How Kubernetes Actually Schedules Your Pod (and What the Network Does After)

DevOps Daily — Thu, 01 Oct 2026 14:23:31 +0000

At some point kubectl apply stops being magic and becomes a question: something decided which node runs this pod, something wired its traffic, and I have no idea what either something did. You can run clusters for a long time in that state. You cannot debug them in it: Pending pods, unreachable Services and mysteriously blocked traffic all live exactly in the parts the tutorials skim.

This is a tour of those parts using three free browser simulators: one for scheduling, one for networking, one for the service mesh layer on top. Each lets you make the decisions yourself, which is the difference between having read about the scheduler and being able to predict it. Disclosure: I help build these; all free, browser-based, no signup, no cluster required.

Part 1: play the scheduler

The K8s Scheduler simulator makes you do the scheduler's job by hand: place workloads (an API, a cache, an ETL job, a search service) onto a small fleet of nodes, against a simplified subset of the real scheduler's constraints, across four levels:

Resource requests: pure bin-packing. Pods declare CPU and memory requests; nodes have capacity; make it fit. Doing this by hand teaches the thing beginners miss: scheduling is based on requests, not actual usage (limits are a kubelet enforcement matter, not a scheduling input), so a cluster can be "full" while its CPUs idle.
Taints, dedicated nodes & anti-affinity: some nodes repel workloads unless tolerated, and some pods must not share a node. This level is where NoSchedule stops being trivia and becomes geometry.
Node selectors & affinity: pods that require labeled nodes, nodeSelector-style. (The game keeps to hard constraints; the real scheduler adds soft "preferred" affinity on top, in its scoring phase.)
Topology spread across zones: distribute replicas to shrink the blast radius of a zone failure, while still fitting everything. This is the level that feels like the real job.

After placing pods by hand, the real scheduler's behavior stops being mysterious: it filters nodes that cannot take the pod (insufficient requests, untolerated taints, failed selectors), scores the survivors (spread, affinity preferences), and binds. When a pod is Pending, the FailedScheduling events in kubectl describe pod report why candidate nodes were rejected, in these same constraint terms; after this game you can read that output as a map instead of an error.

Part 2: follow the packet

The pod is placed; now its traffic needs a path. The Kubernetes Networking & CNI simulator animates the paths that exist in every cluster, and the scenario list doubles as a map of what there is to know: pod-to-pod on the same node, pod-to-pod across nodes, ClusterIP services, NodePort, LoadBalancer, Ingress, service discovery and endpoint slices, egress, and network policies.

Three scenarios are worth the most time:

Same-node versus cross-node: the same pod-to-pod conversation takes a different path depending on placement (which the scheduler from part 1 just decided). Cross-node traffic goes through the CNI's encapsulation or routing, and the simulator lets you switch the dataplane between overlay, routed, and eBPF CNI modes to see the path differences that vendors argue about.
ClusterIP: the scenario that demystifies Services. A Service is not a proxy process sitting somewhere; a node-local dataplane translates the virtual IP into one concrete endpoint as the packet leaves. Watching that translation happen mid-flight is the moment Services make sense.
Network policy: traffic that flowed a second ago stops when a policy selects the pod. The rule underneath: pods are open until a policy selects them, and then they are isolated for that policy's direction (ingress, egress, or both), which is the part everyone otherwise learns during an incident.

The underrated lesson of part 2 is that parts 1 and 2 are coupled: scheduling decisions change packet paths, topology spread changes which conversations cross zones, and the two simulators together explain latency differences no dashboard will.

Part 3: the mesh on top

Once services talk, the next layer of questions is about the talking itself: encryption, retries, rollouts, failure isolation. The Service Mesh simulator tells that story in sequence: the problem (plaintext service-to-service traffic), the mechanism (a sidecar proxy next to each pod), then the features the mechanism buys: automatic mTLS, traffic splitting for safe canary deployments, smart retries, and the circuit breaker that stops a dying service from taking its callers with it.

The sequence matters more than the features. Meshes are usually taught as a feature list, which makes them sound like magic middleware; walked as a story, each capability is obviously "something the proxy in the path can do for you". Whether you need one becomes a cost-benefit question about proxies in request paths, which is the right question.

The debugging payoff

The three parts map onto the three questions of a bad day in production:

Pod is Pending? Part 1: read kubectl describe pod events, in scheduler-constraint terms: requests, taints, affinity, spread.
Pods run but cannot talk? Part 2: walk the path in order: same node or cross-node, Service resolution, then network policy, and check where it dies.
Talking but failing weirdly under load? Part 3: retries, circuit breaking, and whatever the mesh is doing in the path.

A few evenings across the three simulators turns that from a checklist you found in a blog post into paths you have traced yourself. All three are part of 50+ free DevOps games and simulators. For the authoritative version afterwards, the Kubernetes docs on scheduling and Services and networking are the references these simulators are trying to earn you a mental model for.

GitHub Is Removing ssh-rsa Signatures: Find What Breaks Before the November 4 Brownout

DevOps Daily — Thu, 01 Oct 2026 09:00:00 +0000

On September 22, GitHub announced that it will stop accepting two old SSH algorithms: the ssh-rsa signature type, which is RSA with SHA-1, and the diffie-hellman-group-exchange-sha256 key exchange. A laptop with a current OpenSSH will not notice. The clients that break are the ones nobody looks at: an old CI image, a Jenkins agent, a Java tool with an old SSH library, a backup appliance. The first brownout is on November 4, a bad time to find them.

This post covers what changes, which clients and keys are at risk, how to test them, and how to rotate keys.

TLDR

Schedule: new RSA keys need at least 3072 bits from October 14, 2026. Brownouts on November 4 and December 9. Removal on January 13, 2027.
HTTPS remotes are not affected.
Your RSA key can stay if your client signs with rsa-sha2-256 or rsa-sha2-512.
The keys most at risk are old. Since March 2022, only RSA keys added before November 2, 2021 can sign with SHA-1 on GitHub.
Key exchange is the second trap. Besides the one being removed, GitHub offered us only sntrup761, curve25519 and ECDH. A client that supports none of them fails with any key.
Test with ssh -vT git@github.com and read the kex: lines. Add -vvv to see the signature algorithm for your key.

Prerequisites

Git remotes that use SSH (git@github.com:... or ssh://)
Shell access to the machines and images that run Git: CI agents, runners, build containers
The GitHub CLI (gh), with the read:public_key scope for account keys and admin access for deploy keys
OpenSSH's ssh and ssh-keygen

What GitHub is changing, and when

The changelog lists four changes:

Removal of "RSA keys using SHA-1 in SSH (i.e., the ssh-rsa signature type, including ssh-rsa-cert-v01@openssh.com certificates using SHA-1)".
Removal of the key exchange mechanism diffie-hellman-group-exchange-sha256.
"All new RSA SSH keys uploaded after October 14, 2026 must be at least 3072 bits in size, both for signing and authentication." That includes keys you use to sign commits.
A new post-quantum key exchange, mlkem768x25519-sha256, on github.com and on GitHub Enterprise Cloud with data residency, except for the U.S. region.

GitHub SSH change schedule

Sep 22, 2026 changelog published
Oct 14, 2026 new RSA keys 3072+ bits, ML-KEM enabled
Nov 4, 2026 first brownout
Dec 9, 2026 second brownout
Jan 13, 2027 ssh-rsa and DH group exchange removed

Both brownouts cover ssh-rsa and diffie-hellman-group-exchange-sha256. The changelog gives no time of day or duration for them. On GitHub Enterprise Server, everything takes effect in version 3.25, except ML-KEM, which comes in 3.24. GitHub Enterprise Server users of the unauthenticated Git protocol are affected too.

ML-KEM needs nothing from you: older clients "should automatically fall back", GitHub says. OpenSSH added mlkem768x25519-sha256 in 9.9 and made it the default in 10.0, according to its release notes.

Who is actually affected

HTTPS users are not. In GitHub's words: "If your Git remotes start with https://, nothing here will affect you." For SSH users, there are two questions.

1. Can your client sign with SHA-2? As GitHub points out, ssh-rsa is both a key type and a signature type. Every RSA key has key type ssh-rsa, but it can sign with SHA-1 (ssh-rsa), SHA-256 (rsa-sha2-256) or SHA-512 (rsa-sha2-512). The client decides which signature it sends, so the key itself is fine.

In 2021 GitHub announced that from March 15, 2022, "RSA keys uploaded after the cut-off point above will work only with SHA-2 signatures", with November 2, 2021 as the cut-off. So if your RSA key was added after that date and works today, your client already signs with SHA-2. The keys at risk are RSA keys added before November 2, 2021, used by a client that still signs with SHA-1.

2. Does your client share a key exchange with GitHub? This does not depend on your key. On October 1, 2026, GitHub's SSH server offered us this list:

sntrup761x25519-sha512, sntrup761x25519-sha512@openssh.com,
curve25519-sha256, curve25519-sha256@libssh.org,
ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521,
diffie-hellman-group-exchange-sha256, kex-strict-s-v00@openssh.com

Plain Diffie-Hellman groups such as diffie-hellman-group14-sha256 are not offered. When group exchange goes, a client needs sntrup761, curve25519, ECDH or, after October 14, ML-KEM. A client that only does classic Diffie-Hellman fails even with an Ed25519 key.

GitHub lists these minimum versions for RSA with SHA-2 in the default configuration:

Software	Minimum version
OpenSSH	7.2p1
JSch	0.1.66 from the mwiede fork
TeamCity	2021.2.3
Go SSH	0.16.0
libssh2	1.11.0
PuTTY	0.82

Where to look:

Old CI images and build containers. The SSH client is whatever the base image shipped.
Jenkins agents. Command-line Git uses the agent's ssh. An agent set to JGit uses Java SSH code instead, so test it separately.
Java tools on the original JSch (com.jcraft:jsch). The fork GitHub names is published as com.github.mwiede:jsch.
Python tools on Paramiko. It is not in GitHub's table. Its changelog says 2.9.0 (December 2021) added RSA SHA-2 signatures.
libssh2-based tools, such as curl's SFTP and SCP support and many libgit2-based clients.
Windows. Git for Windows ships its own ssh.exe, Windows has a separate built-in OpenSSH, and some setups use PuTTY's plink or TortoiseGit's PuTTY-based plink.
Appliances such as artifact servers and backup tools, which often embed their own SSH library.

Test a client now

Start with ssh -V. Anything older than OpenSSH_7.2 cannot sign with SHA-2, so plan to replace it.

Then connect with verbose output. Key exchange runs before authentication, so no registered key is needed. This is a real run from the Debian 12 machine we wrote this post on, on October 1, 2026:

OpenSSH 9.2p1 against github.com, 2026-10-01

# -F /dev/null ignores ssh_config, no key is offered, the long server-sig-algs line is shortened
$ ssh -F /dev/null -o BatchMode=yes -o PubkeyAuthentication=no -o UserKnownHostsFile=./kh -o StrictHostKeyChecking=accept-new -vT git@github.com 2>&1 | grep -E '^OpenSSH|kex: algorithm|host key algorithm|Server host key|server-sig-algs|Permanently|denied'
OpenSSH_9.2p1 Debian-2+deb12u10, OpenSSL 3.0.22 25 Aug 2026
debug1: kex: algorithm: sntrup761x25519-sha512
debug1: kex: host key algorithm: ssh-ed25519
debug1: Server host key: ssh-ed25519 SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU
Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519-cert-v01@openssh.com,...,rsa-sha2-512,rsa-sha2-256,ssh-rsa>
git@github.com: Permission denied (publickey).

Read three lines:

kex: algorithm: is the agreed key exchange. If it says diffie-hellman-group-exchange-sha256, this client breaks during the brownouts.
kex: host key algorithm: is how GitHub's host key is verified. The changelog words the SHA-1 removal generally, so treat ssh-rsa here as at risk too.
server-sig-algs= lists the signature types GitHub accepts for your key. Today it still ends in ssh-rsa.

For your own test, drop -F /dev/null. Git uses your ssh_config, and a stale KexAlgorithms or HostKeyAlgorithms line there can pin you to the old algorithms.

See both offers. With -vv, OpenSSH prints debug2: KEX algorithms: and debug2: host key algorithms: twice: first your client's offer, then GitHub's. The key exchange list above comes from GitHub's.

See the signature for your key. With -vvv and a key GitHub accepts, OpenSSH 9.2p1 logs a line of this form when it signs:

debug3: sign_and_send_pubkey: signing using rsa-sha2-512 SHA256:<your key fingerprint>

rsa-sha2-512 or rsa-sha2-256 is safe. ssh-rsa breaks. Our machine has no key registered on GitHub, so we confirmed this line against a local test server.

Test without the old algorithms. Remove both from your client's defaults. If this still authenticates, the brownout will not affect this client:

ssh -o KexAlgorithms=-diffie-hellman-group-exchange-sha256 \
    -o HostKeyAlgorithms=-ssh-rsa \
    -o PubkeyAcceptedAlgorithms=-ssh-rsa \
    -T git@github.com

A leading - removes algorithms from the default list. Before OpenSSH 8.5, PubkeyAcceptedAlgorithms was called PubkeyAcceptedKeyTypes.

Know what failure looks like. We forced a key exchange GitHub does not offer, and OpenSSH printed this (GitHub's offer list trimmed):

Unable to negotiate with 140.82.121.4 port 22: no matching key exchange method found. Their offer: sntrup761x25519-sha512,...

With HostKeyAlgorithms=ssh-dss, it printed no matching host key type found. Search CI logs for these strings on November 4. Libraries word their errors differently.

List what the binary supports. ssh -Q kex lists key exchanges and ssh -Q sig lists signature algorithms (added in OpenSSH 7.9). For the settings your config actually applies to GitHub, run ssh -G github.com and read kexalgorithms, hostkeyalgorithms and pubkeyacceptedalgorithms.

Test the clients that are not your shell's ssh

Which ssh does Git run? core.sshCommand, GIT_SSH and GIT_SSH_COMMAND can point Git at another program. Ask Git:

GIT_TRACE=1 git ls-remote git@github.com:your-org/your-repo.git 2>&1 | grep run_command

On our machine the run_command line contained ssh -o SendEnv=GIT_PROTOCOL git@github.com 'git-upload-pack ...', so Git used the ssh on the path.

Java. Find the original JSch in your dependency tree:

mvn dependency:tree -Dincludes=com.jcraft:jsch
./gradlew dependencyInsight --dependency com.jcraft --configuration runtimeClasspath

The fork's README shows how to exclude com.jcraft:jsch when it arrives as a transitive dependency.

Python. python3 -m pip show paramiko prints the installed version.

libssh2. curl -V shows the libssh2 version curl links against. On our Debian 12 machine it printed libssh2/1.10.0, below GitHub's 1.11.0. We did not test whether it fails against GitHub, and distributions sometimes backport fixes, so test the tool rather than trusting the number.

Go. GitHub does not name the module behind "Go SSH". If it means golang.org/x/crypto, check its version in go.mod.

Windows. In PowerShell, Get-Command ssh shows which ssh.exe comes first on the path. Also check git config --show-origin --get core.sshCommand and GIT_SSH. If they point at plink, compare its version with PuTTY 0.82.

Audit your keys

ssh-keygen -l -f ~/.ssh/id_rsa.pub prints a key's size in bits, its fingerprint and its type, such as (RSA) or (ED25519). With -f - it reads keys from standard input.

Account keys. This lists every authentication key on your account with its upload date, size and type:

# Needs the read:public_key scope: gh auth refresh -h github.com -s read:public_key
gh api /user/keys --paginate --jq '.[] | [.created_at[0:10], .title, .key] | @tsv' |
  while IFS=$'\t' read -r created title key; do
    size_type=$(printf '%s\n' "$key" | ssh-keygen -l -f - | awk '{print $1, $NF}')
    printf '%s %-14s %s\n' "$created" "$size_type" "$title"
  done

RSA keys dated before 2021-11-02 are the only ones GitHub still lets sign with SHA-1. Check the clients that use them first.

Deploy keys. The deploy keys API returns created_at and last_used. This loop needs admin access to each repository and silently skips any it cannot read, so check the count:

ORG=your-org
gh repo list "$ORG" --limit 1000 --json nameWithOwner --jq '.[].nameWithOwner' |
  while read -r repo; do
    gh api "repos/$repo/keys" --paginate \
      --jq ".[] | [\"$repo\", .id, .created_at[0:10], (.last_used // \"never\")[0:10], .title, .key] | @tsv" 2>/dev/null
  done |
  while IFS=$'\t' read -r repo id created used title key; do
    size_type=$(printf '%s\n' "$key" | ssh-keygen -l -f - | awk '{print $1, $NF}')
    printf '%s\t%s\tadded %s\tused %s\t%s\t%s\n' "$repo" "$id" "$created" "$used" "$size_type" "$title"
  done

A key that was never used is a candidate for deletion, not rotation.

Machine users. Public authentication keys need no token:

curl -s https://github.com/your-machine-user.keys | ssh-keygen -l -f -

For upload dates, run the account key audit with the machine user's token.

Rotate to Ed25519

GitHub recommends "an Ed25519 key whenever possible". For an account key:

# 1. Create the key. Add -N '' only for unattended CI keys.
ssh-keygen -t ed25519 -C "ci-runner-2026-10" -f ~/.ssh/id_ed25519_github

# 2. Register it. If the token lacks the scope, gh prints the refresh command.
gh ssh-key add ~/.ssh/id_ed25519_github.pub --title "ci-runner 2026-10"

# 3. Prove the new key works on its own.
ssh -i ~/.ssh/id_ed25519_github -o IdentitiesOnly=yes -T git@github.com

Then point Git at it in ~/.ssh/config (our post on using a specific SSH key for Git has other ways):

Host github.com
  IdentityFile ~/.ssh/id_ed25519_github
  IdentitiesOnly yes

When nothing uses the old key, find its ID with gh ssh-key list and remove it with gh ssh-key delete <id>.

For deploy keys, gh repo deploy-key add key.pub --title "deploy 2026-10" -R your-org/your-repo adds a read-only key, and --allow-write grants push access. Its help text warns that the key is "associated with the current authentication token" and is removed if that token is de-authorized. Add long-lived deploy keys in the repository settings instead.

Warning

A new key type does not fix a key exchange problem. If a client's only key exchange in common with GitHub is diffie-hellman-group-exchange-sha256, it fails on January 13 with any key. Upgrade the client or library first.

Where Ed25519 is not supported:

ECDSA. GitHub says "All Ed25519 and ECDSA keys we support are strong, secure, and will continue to work for the indefinite future." Use ssh-keygen -t ecdsa -b 256.
RSA with 3072 bits or more, if another service needs RSA: ssh-keygen -t rsa -b 4096. The client must still sign with SHA-2.
Old key parsers. Since OpenSSH 7.8, ssh-keygen writes its own private key format. Some older libraries only read PEM, which -m PEM produces for RSA and ECDSA keys.

Checklist before November 4

CI images and agents

List every image, runner and agent that runs Git over SSH, including container jobs.
Run ssh -V in each. Replace anything older than OpenSSH 7.2.
Run the test without old algorithms inside each image, with the job's credentials.
Check /etc/ssh/ssh_config, /etc/ssh/ssh_config.d/ and baked-in ~/.ssh/config files for KexAlgorithms, HostKeyAlgorithms, PubkeyAcceptedAlgorithms and PubkeyAcceptedKeyTypes.
Find JSch, Paramiko, libssh2 and Go SSH in your build tools.

Deploy keys and machine users

Run the deploy key audit for each organization and delete unused keys.
For RSA keys added before November 2, 2021, test the client that uses them, or switch to Ed25519.
Audit each machine user's keys with its own token, and test from the host that uses it.

On the day

Watch CI for Unable to negotiate and Permission denied (publickey).
Treat any failure as a real finding, even if it stops after the brownout.

What we could not verify

Brownout timing. The changelog gives dates, not times or durations.
Existing small RSA keys. The 3072-bit rule covers "new RSA SSH keys uploaded after October 14, 2026". The changelog does not say existing 2048-bit keys stop working. We assume deploy keys and re-uploaded old keys count as new uploads, but it does not say so.
Libraries. We did not test JSch, Paramiko, libssh2, Go or PuTTY. The table is GitHub's.
A real SHA-1 signature. We had no RSA key from before November 2021 to test with.
Server lists. These are what GitHub offered on October 1, 2026. They can change.

Summary

GitHub removes RSA with SHA-1 signatures and one Diffie-Hellman key exchange on January 13, 2027, with brownouts on November 4 and December 9. HTTPS remotes are not affected. Your RSA key can stay, but the client must sign with SHA-2 and share a modern key exchange with GitHub.

Before November 4, test every place that runs Git over SSH with the old algorithms removed, and audit account keys, deploy keys and machine users. Move to Ed25519 where you can, and to ECDSA or 3072-bit RSA where you cannot. For another client-side SSH risk in some of the same libraries, see our post on libssh2 CVE-2026-55200.

Originally published at devops-daily.com.

GitHub Actions Removed Node 20. Find Every node20 Action You Still Run

DevOps Daily — Thu, 01 Oct 2026 09:00:00 +0000

On September 23, 2026, GitHub removed Node 20 from GitHub Actions runners. The final changelog post is short: runners now use Node 24 for JavaScript actions, and the temporary opt-out is gone. What it does not spell out is what happens to the actions whose action.yml still says node20. They do not stop. The runner starts them on Node 24 instead, adds a warning to the job, and moves on. If the action works on Node 24, you get a yellow annotation. If it does not, the step fails or misbehaves, and there is no setting left that brings Node 20 back.

That makes the change quiet, which is the problem. This post shows how to list every action your workflows use, read the runtime each one declares, and find the node20 actions hidden behind SHA pins and composite actions. Then it covers what to do about each one, and the self-hosted runner hosts that GitHub no longer supports.

TLDR

Node 20 is gone from GitHub Actions runners as of September 23, 2026. JavaScript actions run on Node 24, and ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION no longer brings Node 20 back.
node20 actions are not rejected. The runner forces them onto Node 24 and adds a job warning that names them. Jobs we checked on September 28 passed with that warning.
The risk is the action that breaks on Node 24, because you can no longer fall back.
Floating major tags do not save you. On October 1, actions/checkout@v4, actions/cache@v4, actions/setup-node@v4 and actions/upload-artifact@v5 all still declared node20.
SHA pins inside other people's composite actions are invisible to Dependabot, because the pin lives in a repository you do not own.
A 100-line Python script lists every uses: reference in a repository, follows composite actions, and prints each one's runs.using.
Self-hosted runners on macOS 13.4 or older, or on ARM32, are no longer supported.

Prerequisites

A repository with GitHub Actions workflows, checked out locally
The GitHub CLI (gh), logged in. The script uses it to read action.yml files from other repositories, including private ones your account can read.
Python 3.8 or newer
For the runner section: admin access to list self-hosted runners, or shell access to the runner hosts

What GitHub removed, and when

GitHub announced the deprecation on September 19, 2025, because Node 20 reached end of life in April 2026. The plan moved several times, and the editor's notes on that post record each move. The final timeline was:

Runner v2.328.0 added Node 24 next to Node 20, with Node 20 still the default. Setting FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true let you test Node 24 early.
June 16, 2026: runners started using Node 24 by default. ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true let you opt back into Node 20.
September 23, 2026: Node 20 was removed, and that opt-out stopped working.

The September 23 post says: "This is the final notification that Node 20 is no longer available on GitHub Actions runners. Runners now use Node 24 for JavaScript actions. The temporary ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION opt-out is no longer available." It applies to github.com and GitHub with Data Residency. Maintainers should set runs.using to node24 and release; workflow authors should move to versions that support Node 24.

What happens to a node20 action now

The changelog says runners now use Node 24 for JavaScript actions. The runner source shows what that means for an action that declares node20. In the final phase, the runner picks Node 24 for any action that declares node20, whatever environment variables you set. Actions that declare node12 or node16 are first mapped to node20, so they end up on Node 24 too. At the end of the job, the runner adds a warning that lists them.

We looked at the annotations of three jobs that ran on GitHub-hosted runners on September 28, in an open-source ebook repository that still pins old actions. All three jobs passed. One of them carried this warning:

Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/cache@v4, actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/

Another job used actions/checkout@v2, which declares node12, and its warning listed actions/checkout@v2 as a Node.js 20 action forced onto Node 24. That matches the mapping in the source.

What the runner does with an old JavaScript action

runs.using: node12 or node16 mapped to node20 first
runs.using: node20 action.yml
Runner picks Node 24 no opt-out since Sept 23

Outcomes:

Code works on Node 24 step passes, job gets a warning
Code breaks on Node 24 step fails, no way back to Node 20

What can break? Node 24 is two major versions after Node 20. The Node.js 24.0.0 release notes list removals such as tls.createSecurePair and fs.Dirent's path property, and runtime deprecations such as url.parse(). Whether an action hits one of these depends on its code and its dependencies. Until you have run it on Node 24, treat a node20 action as untested.

Why SHA pins and unmaintained actions are the main risk

Pinning actions to a full commit SHA is good supply-chain practice. It also freezes the runtime. actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 is v4.2.2, and it will declare node20 forever. The pin works as intended. It hides the update.

Floating major tags do not help much either. The first-party actions we checked moved to Node 24 in new major versions, so the old major tags stay on Node 20. We read the action.yml of each tag on October 1:

Action	Still `node20`	First major on `node24`
`actions/checkout`	`v4`	`v5`
`actions/setup-node`	`v4`	`v5`
`actions/cache`	`v4`	`v5`
`actions/upload-artifact`	`v4` and `v5`	`v6`

Note upload-artifact: v5 still declares node20. A newer major does not always mean Node 24, so read the file.

Dependabot helps less than you might expect. Per GitHub's docs, version updates for actions only run when dependabot.yml lists the github-actions ecosystem. Dependabot only supports the owner/repo@ref syntax, ignores actions and reusable workflows referenced by a local path, and does not support docker:// references. It also only edits files in your repository. When a third-party composite action pins a node20 action inside its own action.yml, no pull request in your repository can fix it.

That case is easy to find in the wild. The latest release of dominikh/staticcheck-action, v1.4.1 from March 12, 2026, is a composite action. Inside, it pins actions/cache to the commit for v4.3.0, which declares node20. Your workflow says dominikh/staticcheck-action@v1.4.1, and nothing in it mentions Node.

Then there are unmaintained actions. If an action has not had a release in years, no node24 version is coming. You need a plan for it, not a version bump.

How to find every node20 action

The method is simple:

List every uses: value in .github/workflows/*.yml and in any action.yml in the repository.
Split each value into owner, repository, optional path, and ref.
Read that action's action.yml or action.yaml at that ref, and look at runs.using.
If it says composite, repeat for the steps inside it.

You can do step 3 by hand with gh or with raw.githubusercontent.com:

read runs.using by hand

$ gh api "repos/actions/checkout/contents/action.yml?ref=v4" --jq .content | base64 -d | grep using:
  using: node20
$ curl -s https://raw.githubusercontent.com/actions/checkout/v5/action.yml | grep using:
  using: node24

The gh route works for private repositories and accepts any ref: a tag, a branch, or a SHA. For every reference in every workflow, use the script below.

A quick signal from job annotations

If a workflow ran recently, its jobs already carry the runner's warning. You can read it without opening the web UI:

# Job IDs for one run
gh run view RUN_ID --json jobs --jq '.jobs[].databaseId'

# The Node 20 warning for one job, if it has one
gh api repos/OWNER/REPO/check-runs/JOB_ID/annotations \
  --jq '.[] | select(.message | startswith("Node.js 20")) | .message'

This only covers what ran. A release workflow that runs once a quarter will not show up until it fails. The scan below reads what the workflows declare instead.

The script

It reads workflows and local actions from disk and fetches everything else through the GitHub API:

#!/usr/bin/env python3
"""Print the runtime (runs.using) of every action a repository's workflows use.

Usage: find-node20-actions.py [path-to-repo] (needs python3 and a logged-in gh)
Exit codes: 0 all clear, 1 node12/16/20 actions found, 2 some refs could not be checked.
"""
import base64
import os
import pathlib
import re
import subprocess
import sys

USES = re.compile(r"""^\s*(?:-\s+)?uses:\s*['"]?([^'"\s#]+)""")
USING = re.compile(r"""^\s+using:\s*['"]?([A-Za-z0-9_-]+)""")
OLD = {"node12", "node16", "node20"} # all of these now run on Node 24
PROBLEMS = {"not found", "unparsed", "unknown"}

root = pathlib.Path(sys.argv[1] if len(sys.argv) > 1 else ".").resolve()
seen = {} # ref -> (runtime, text), so each action is fetched and walked once
found_old = found_problem = False

def uses_in(text):
    return [m.group(1) for line in text.splitlines() if (m := USES.match(line))]

def runtime_of(text):
    for line in text.splitlines():
        if m := USING.match(line):
            return m.group(1)
    return "unknown"

def read_remote(owner, repo, path, ref):
    for name in ("action.yml", "action.yaml"):
        file = f"{path}/{name}" if path else name
        r = subprocess.run(
            ["gh", "api", f"repos/{owner}/{repo}/contents/{file}?ref={ref}", "--jq", ".content"],
            capture_output=True, text=True,
        )
        if r.returncode == 0:
            return base64.b64decode(r.stdout).decode()
    return None

def resolve(ref):
    """Return (runtime, action.yml text or None) for one uses: value."""
    if ref.startswith("docker://"):
        return "docker image", None
    if re.search(r"\.ya?ml(@|$)", ref):
        return "reusable workflow", None # scan that workflow's repo too
    if ref.startswith("./"):
        for name in ("action.yml", "action.yaml"):
            file = root / ref / name
            if file.is_file():
                text = file.read_text()
                return runtime_of(text), text
        return "not found", None
    m = re.fullmatch(r"([^/@]+)/([^/@]+)(?:/([^@]+))?@(.+)", ref)
    if not m:
        return "unparsed", None
    owner, repo, path, version = m.groups()
    text = read_remote(owner, repo, path or "", version)
    if text is None:
        return "not found", None
    return runtime_of(text), text

def walk(source, text):
    global found_old, found_problem
    for ref in uses_in(text):
        first = ref not in seen
        if first:
            seen[ref] = resolve(ref)
        runtime, action_text = seen[ref]
        found_old |= runtime in OLD
        found_problem |= runtime in PROBLEMS
        flag = "!!" if runtime in OLD else " "
        print(f"{flag} {source:<38} {ref:<52} {runtime}")
        # A remote composite action can wrap a node20 action. Local ones are scanned as files below.
        if first and runtime == "composite" and action_text and not ref.startswith("./"):
            walk(ref, action_text)

files = sorted(root.glob(".github/workflows/*.y*ml"))
for dirpath, dirnames, filenames in os.walk(root):
    dirnames[:] = sorted(d for d in dirnames if d not in ("node_modules", ".git"))
    files += [pathlib.Path(dirpath, n) for n in sorted(filenames) if n in ("action.yml", "action.yaml")]
for f in files:
    text = f.read_text()
    rel = f.relative_to(root)
    if f.name.startswith("action."):
        runtime = runtime_of(text)
        found_old |= runtime in OLD
        found_problem |= runtime in PROBLEMS
        print(f"{'!!' if runtime in OLD else ' '} {str(rel):<38} {'(this action)':<52} {runtime}")
    walk(str(rel), text)

sys.exit(2 if found_problem else 1 if found_old else 0)

What it handles:

./local-action paths are read from disk, relative to the repository root.
docker:// references are reported as Docker images, which do not use the runner's Node.
Composite actions in other repositories are followed, so a node20 action two levels down still shows up. Local ones are scanned as files.
action.yml and action.yaml are both tried, in that order. GitHub's metadata docs allow either name.
Subdirectory actions such as github/codeql-action/init@v4.38.2 resolve to that path in the repository.
Reusable workflows are labeled, not followed. Scan their repositories separately.

A reference it cannot read prints not found and makes the exit code 2, so an expired token or a deleted repository never reads as a clean result.

What we ran

We built a test repository whose workflow mixes every case: a SHA-pinned action, two local actions (one composite, one JavaScript with an action.yaml), a docker:// image, two third-party actions, a subdirectory action, and a reusable workflow. The octo-org reference is a made-up name, which the script labels without fetching:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
      - uses: ./.github/actions/setup # composite, uses actions/setup-node@v4
      - uses: ./.github/actions/legacy-js # action.yaml, runs.using: 'node20'
      - uses: docker://alpine:3.20
      - uses: JS-DevTools/npm-publish@v4.1.5
      - uses: dominikh/staticcheck-action@v1.4.1
      - uses: github/codeql-action/init@v4.38.2
  deploy:
    uses: octo-org/shared/.github/workflows/deploy.yml@main

Here is the run, on October 1, 2026:

scan the test repository

$ python3 find-node20-actions.py demo-repo; echo "exit code: $?"
!! .github/workflows/ci.yml actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 node20
   .github/workflows/ci.yml ./.github/actions/setup composite
!! .github/workflows/ci.yml ./.github/actions/legacy-js node20
   .github/workflows/ci.yml docker://alpine:3.20 docker image
   .github/workflows/ci.yml JS-DevTools/npm-publish@v4.1.5 node24
   .github/workflows/ci.yml dominikh/staticcheck-action@v1.4.1 composite
   dominikh/staticcheck-action@v1.4.1 actions/setup-go@4b73464bb391d4059bd26b0524d20df3927bd417 node24
!! dominikh/staticcheck-action@v1.4.1 actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 node20
   .github/workflows/ci.yml github/codeql-action/init@v4.38.2 node24
   .github/workflows/ci.yml octo-org/shared/.github/workflows/deploy.yml@main reusable workflow
!! .github/actions/legacy-js/action.yaml (this action) node20
   .github/actions/setup/action.yml (this action) composite
!! .github/actions/setup/action.yml actions/setup-node@v4 node20
exit code: 1

Rows marked !! need work. The actions/cache row is the one Dependabot cannot fix: it comes from inside staticcheck-action.

We also ran the script on two real repositories. On this site's own repository it printed 20 rows across six workflows, all node24 or docker, and exited 0. On the ebook repository from earlier, it flagged actions/checkout@v4, actions/checkout@v2 (node12), and the actions/cache@v4 inside a composite build action. Those are the same three actions that GitHub's job annotations listed for its September 28 runs.

To scan a whole organization, run the script in each checked-out repository. Remember that it only sees the branch you have checked out.

Fix each affected action

For each !! row you have three options.

Upgrade to a node24 release. This is the usual fix. Find the newest release, read its action.yml to confirm node24, and update the reference. For SHA pins, update the SHA and the version comment together, so Dependabot and humans can still read it. Check the release notes for breaking changes, because the Node 24 releases were often new major versions with other changes too.

Replace it. If the action is unmaintained, switch to a maintained one, or drop it for a run: step. Many small actions wrap one CLI command, and gh release create or aws s3 sync in a run: step has no Node runtime to go stale.

Fork it and bump runs.using. For an action with no node24 release, fork it, change runs.using to node24, run its tests on Node 24, rebuild any bundled dist/ folder, and pin your fork by SHA. The runner already runs the action on Node 24, so the edit alone only removes the warning. The value is in the testing and in owning the fix. You also own the fork's security updates now.

For composite actions you do not own, such as the staticcheck-action case, the fix belongs upstream. Open an issue or a pull request, and fork in the meantime if the wrapped action breaks.

Internal JavaScript actions

Your own actions need the same change, and nobody else will make it. To find them across an organization, GitHub code search works:

gh search code node20 --owner YOUR_ORG --filename action.yml
gh search code node20 --owner YOUR_ORG --filename action.yaml

Search for the bare word. In our tests, the phrase "using: node20" returned nothing, even in an organization where the bare word found files that contain exactly that line. Code search only covers default branches, and the word can also match comments and test fixtures, so confirm each hit with the script. For each real one:

Set runs.using: node24 in its action.yml.
Run its tests on Node 24, and set the same version in its own CI.
Rebuild any bundled output, such as a dist/ folder built with ncc.
Publish a new tag, and move the major tag if your consumers use one.
Update the repositories that pin the old SHA. The script above finds them.

Self-hosted runners need a runner version that knows node24. The announcement names v2.328.0 as the release that added it. GitHub now enforces much newer runner versions anyway, as we covered in our post on the self-hosted runner version window.

Self-hosted runner hosts that lose support

The September 23 post also says: "Node 24 is incompatible with macOS 13.4 and earlier, and it doesn't officially support ARM32. Self-hosted runners using these operating systems or architectures are no longer supported." The Node.js 24.0.0 release notes agree: the minimum macOS version went up to 13.5, and armv7 support was downgraded to experimental.

The runner source has a kill switch for this. On Linux ARM32, when GitHub turns it on, a JavaScript action step fails with "Linux ARM32 runners are no longer supported. Please migrate to a supported platform." That message is from the runner source. We have no ARM32 runner and did not see it in a real job.

To find these hosts, start with the API:

gh api repos/OWNER/REPO/actions/runners \
  --jq '.runners[] | [.name, .os, ([.labels[].name] | join(","))] | @tsv'

Use orgs/YOUR_ORG/actions/runners for organization runners. Do not trust the labels alone. GitHub's docs note that when default labels such as x64 are set with the configuration script, GitHub "does not validate that the runner is actually using that operating system or architecture." On the hosts themselves, uname -m shows the architecture (armv7l or armv6l means ARM32), and sw_vers -productVersion shows the macOS version.

There is no Node-only fix for these hosts. Move ARM32 runners to arm64 hardware or an arm64 OS image, and upgrade old Macs or retire them. A workflow with only run: steps and Docker actions needs no JavaScript action, but almost every workflow starts with actions/checkout.

Caveats

The script is a line scanner, not a YAML parser. It can miss uses: in flow-style YAML, and it can pick up a line that starts with uses: inside a multi-line run: block. It takes the first indented using: line as the runtime.
It reads one branch. Workflows that exist only on other branches are not scanned.
It does not test anything. A node20 row means the action declares Node 20. Whether it works on Node 24 is a separate question.
Reusable workflows from other repositories are not followed.
Our runtime check is one snapshot. The tag table and the scan output are from October 1, 2026. Tags move, and maintainers ship releases.
We did not test GitHub Enterprise Server. The September 23 post names github.com and GitHub with Data Residency only.

Summary

Node 20 left GitHub Actions on September 23, 2026, and the opt-out went with it. Actions that declare node20, node16 or node12 now run on Node 24, with a warning that names them. The ones that still work are easy to ignore, and the first one that breaks has no fallback.

Read the job annotations for a quick signal, then scan what your workflows declare: every uses: reference, resolved to its action.yml, with composite actions followed. Watch SHA pins, old major tags, and node20 actions pinned inside third-party composite actions, because no automated update reaches those. Upgrade, replace, or fork each one, move your own JavaScript actions to node24, and retire any self-hosted runner on ARM32 or macOS 13.4 or older.

Originally published at devops-daily.com.

Free Egress Has Fine Print: S3 vs R2, B2, Wasabi and More

DevOps Daily — Wed, 30 Sep 2026 09:00:00 +0000

Serving 50 TB a month out of an S3 bucket in us-east-1 costs about $4,379 at list price, almost all of it data transfer. The same month on Cloudflare R2 costs about $62. That gap is why "zero egress" object storage is a whole product category now: Cloudflare R2, Backblaze B2, Wasabi, Tigris, and bundles from DigitalOcean, Hetzner and others.

The cheapest number in our comparison, though, is one you cannot rely on. Wasabi comes out at $15.60 for that 50 TB month, and that month is outside Wasabi's free-egress policy. Every provider in this category has fine print, and it is different for each one: a ratio to your stored data, a fair-use policy, a price per read, a minimum storage duration, a bundle that runs out. We put nine storage offerings from eight providers into a small calculator with their September 2026 prices and the main fine print applied, and ran four workloads through it.

The-DevOps-Daily/object-storage-egress-calc on GitHub

TLDR

For egress-heavy workloads, the zero-egress providers win by one or two orders of magnitude. Our 50 TB media month: R2 $62, Hetzner $76, Tigris $90, against $4,379 on S3 at list price.
Wasabi's free egress is a policy, not a price. Egress should stay at or below the data you store. Above that there is no overage rate; Wasabi reserves the right to limit or suspend the service.
Backblaze B2 is free only up to 3x your stored data, then $0.01/GB, unless you go through a partner CDN.
R2 egress is free, but reads are not. In the media workload, GET requests were about half of the R2 bill.
Minimum storage durations beat egress for backups. With 30-day retention, Wasabi's 90-day minimum triples its storage bill and puts it next to S3 at the bottom of the table.
AWS changed the math in November 2025 with flat-rate CloudFront plans. A direct per-GB comparison against S3 is no longer the whole story.

Prerequisites

A rough idea of your workload per month: data stored, data served to the internet, number of GET and PUT requests, and how long objects live.
Python 3, if you want to run the calculator. It has no dependencies.

The headline prices

This is what the pricing pages lead with, as of 28 September 2026. Every price comes from the provider's own pages. The repository's providers.json links the sources for every price the calculator uses, and other claims link their sources where they appear. Scenario totals are the calculator's estimates, not quotes.

Provider	Storage per GB-month	Internet egress
AWS S3 Standard (us-east-1)	$0.023	100 GB free per account, then $0.09/GB, falling to $0.05/GB above 150 TB
Cloudflare R2 Standard	$0.015	Free
Cloudflare R2 Infrequent Access	$0.01	Free, but $0.01/GB retrieval on every read
Backblaze B2	$6.95 per TB	Free up to 3x stored, then $0.01/GB
Wasabi	$7.99 per TB	Free, subject to policy
Tigris Standard	$0.02	Free
DigitalOcean Spaces	$5/month for 250 GiB, then $0.02/GiB	1,024 GiB included, then $0.01/GiB
Hetzner Object Storage	$7.99/month for about 1 TB	About 1 TB included, then $1.20/TB
Storj Standard	$7 per TB	$7 per TB

For context, the other two big clouds are in the same place as S3: Google Cloud Storage lists $0.12/GiB for the first 10 TiB to most destinations, and Azure includes 100 GB a month, then charges $0.087/GB for the next 10 TB from North America or Europe over its Premium Global Network.

Several of these prices are new. Backblaze went from $6 to $6.95/TB on 1 May 2026 and made standard API calls free (event notifications still cost). Wasabi went from $6.99 to $7.99/TB on 1 July. Hetzner raised its base price from EUR 4.99 to EUR 6.49 on 1 April. Storj moved to its "Simplified" pricing on 1 July. Comparisons written in 2025 are out of date.

Four workloads

The calculator prices one month for each provider and prints a note wherever the workload breaks a rule. Here are the four workloads from scenarios.sh, with the results exactly as recorded in runs/scenarios.txt:

Provider	Downloads site	Backups	Media	Side project
Workload	500 GB stored, 5 TB egress, 20M GETs	10 TB stored, 200 GB egress, 30-day retention	2 TB stored, 50 TB egress, 100M GETs	50 GB stored, 200 GB egress, 1M GETs
AWS S3 Standard	$460.55	$244.04	$4,379.20	$10.60
Cloudflare R2 Standard	$10.95	$149.85	$62.25	$0.60
Cloudflare R2 IA	$73.09	$111.09	$610.90	$3.49
Backblaze B2	$38.41	$69.43	$453.83	$0.78
Wasabi	$7.99, outside policy	$234.00	$15.60, outside policy	$7.99, outside policy
Tigris Standard	$19.85	$204.85	$90.30	$1.35
DigitalOcean Spaces	$49.76	$200.00	$529.76	$5.00
Hetzner Object Storage	$12.69	$87.69	$75.55	$7.99
Storj Standard	$38.50	$71.40	$364.00	$5.00

Media: 2 TB stored, 50 TB egress, 100M GETs per month

	Value
Wasabi (outside policy)	15.6$
Cloudflare R2 Standard	62.25$
Hetzner	75.55$
Tigris Standard	90.3$
Storj Standard	364$
Backblaze B2	453.83$
DigitalOcean Spaces	529.76$
Cloudflare R2 IA	610.9$

List prices on 28 September 2026, from the calculator in the linked repository. Wasabi's figure breaks its free-egress policy (egress above stored data), so it is not a price you can rely on. AWS S3 Standard for the same month is $4,379.20 at list price, off this scale.

Two things stand out. The order changes completely between workloads: Wasabi is cheapest for media and second most expensive for backups. And the reasons it changes are all in the fine print, not in the headline egress price. The rest of this post goes through that fine print, one rule at a time.

Wasabi: free egress is a policy, not a price

Wasabi's pricing FAQ is explicit: if your monthly egress is at or below your active storage, your use case "is a good fit" for free egress. If it is above, it "is not a good fit", and if that happens "on a regular basis, we reserve the right to limit or suspend your service." There is no overage price to pay instead.

So Wasabi is a poor fit for workloads that regularly serve more than they store, such as downloads, media and most public assets, however good the number looks. The calculator prints the price with a note rather than hiding the row, because the policy is judged over time and a single heavy month is not the same as a pattern.

Wasabi also has three minimums: a 1 TB monthly minimum (the side project pays $7.99 for 50 GB), a 4 KB minimum object size, and a 90-day minimum storage duration, which is the next rule.

Minimum storage duration: why Wasabi loses on backups

A backup bucket with 30-day retention deletes every object after 30 days. On a provider with a 90-day minimum, each of those objects is billed as if it stayed 90 days. In a steady state that is three times the storage you actually hold, and it is why Wasabi's backup month costs $234.00 instead of about $78.

Backups: 10 TB stored, 30-day retention, 200 GB restored

	Value
Backblaze B2	69.43$
Storj Standard	71.4$
Hetzner	87.69$
Cloudflare R2 IA	111.09$
Cloudflare R2 Standard	149.85$
DigitalOcean Spaces	200$
Tigris Standard	204.85$
Wasabi	234$
AWS S3 Standard	244.04$

List prices on 28 September 2026, from the calculator. Wasabi bills each object for 90 days, so 30-day retention is charged at about three times the stored data. Storj and R2 Infrequent Access have 30-day minimums, which 30-day retention just meets.

The minimums across the providers here: Wasabi 90 days, and overwrites count as deletes. Storj 30 days. R2 Infrequent Access, Tigris Infrequent Access and DigitalOcean Cold Storage 30 days, Tigris Archive 90 days. None for S3 Standard, R2 Standard, B2 or Tigris Standard. Any bucket that churns, such as a build cache, CI artifacts or daily exports, pays the minimum on every object, whatever the egress price.

Backblaze B2: free up to 3x what you store

B2's egress is free "up to 3x their average monthly storage", measured in byte-hours over the month, then $0.01/GB. It is unlimited only when you download to or through partner CDNs and compute providers (the page names Fastly, Cloudflare, bunny.net, CacheFly, CoreWeave, Equinix Metal, Vultr and phoenixNAP), or on B2 Overdrive, which needs a multi-petabyte commitment.

That is why B2 does well on backups and less well on the downloads site: 500 GB stored gives 1.5 TB of free egress, and the other 3.5 TB costs $35. The overage rate is still a ninth of S3's, and a partner CDN in front removes it.

Cloudflare R2: egress is free, reads are not

R2 charges nothing for egress, with no ratio and no policy on the pricing page. It does charge for operations: every GET and HEAD is a Class B operation at $0.36 per million after the free 10 million a month. In the media workload, 100 million GETs cost $32.40 of the $62.25 total, more than the storage.

R2 Infrequent Access is a different product. It adds a $0.01/GB retrieval fee on every read or copy, a 30-day minimum, and no free tier. That is fine for backups, and after S3 it is the most expensive choice here for anything people download: $610.90 for the media month, because 50 TB of reads is $500 of retrieval.

Two more details. The r2.dev public URL is rate-limited and meant for development; public buckets in production should use a custom domain. And Cloudflare's service terms say that, unless you are an Enterprise customer, serving video and other large files through its CDN requires one of its paid services, such as the Developer Platform (which includes R2), Images or Stream. Check those terms before you put a large-file origin hosted elsewhere behind Cloudflare.

Bundles: a fixed amount of free egress, then per GB

Some providers sell a monthly bundle instead of a zero price:

DigitalOcean Spaces: $5 a month includes 250 GiB of storage and 1,024 GiB of outbound transfer shared across all buckets, then $0.02/GiB stored and $0.01/GiB transferred. The built-in CDN is included, and its traffic counts against the same allowance. Transfer from Spaces to Droplets in the same datacenter group is free, so a Spaces bucket next to DigitalOcean compute only pays for what leaves for the internet.
Hetzner: $7.99 (EUR 6.49) a month includes about 1 TB of storage and 1 TB of egress, accrued hour by hour, so a 30-day month holds 1.08 TB of egress. Extra egress is only $1.20/TB, which is why Hetzner is right behind R2 in the media workload. You pay the base price for any hour you have a bucket, even an empty one, and unused quota does not carry over.
Storj: no bundle and no free egress. $7/TB for storage and $7/TB for egress from the first byte, with a $5 minimum invoice (accounts that pay in USDC are exempt).

Bundles are easy to predict and cheap for small projects. DigitalOcean's $5 covers the side project workload in full. What matters is where the included transfer runs out: DigitalOcean's overage is $0.01 per GiB, the same number as B2's $0.01 per GB above 3x, and it is the main line on its media bill.

Small objects pay more than their size

Several providers bill a minimum object size: Hetzner 64 KB, Storj 50 kB, Wasabi 4 KB, DigitalOcean 4 KiB (128 KiB for Cold Storage, on storage and on every read). A bucket of 10 KB thumbnails is billed at 6.4 times its real size on Hetzner. The calculator does not model this, so if your average object is small, check it against the list before you trust the result. R2, Tigris, S3 Standard and B2 state no minimum.

AWS: the free 100 GB is per account, and CloudFront changed the math

AWS's 100 GB of free internet egress is shared by the whole account, across all services and Regions, and the volume tiers also count all services together. A bucket in an account that also runs busy EC2 instances may never see the free 100 GB.

The bigger change is CloudFront. Transfer from S3 to CloudFront has long been free, and since 18 November 2025 CloudFront has flat-rate plans with "no overage charges": Pro is $15 a month for a 50 TB allowance and 10 million requests, Business $200 for 50 TB and 125 million requests. Above the allowance, AWS says the first spike up to 3x will not affect service that month, but if you keep exceeding it without upgrading, "your traffic delivery might be adjusted", for example served from fewer or more distant edge locations.

We did not model this, because the bill then depends on cache hit rates and on how AWS treats sustained overuse. But it means "S3 costs $4,379 for 50 TB" is only true if you serve straight from the bucket. With S3 behind a flat-rate plan, the AWS number for a cache-friendly workload moves toward storage plus the plan fee, and that belongs in the same comparison as R2.

Bytes you pay for but never deliver

AWS bills the bytes it sent when a client aborts a download, with its own example of 3 GB billed for a request cut off at 2 GB. DigitalOcean bills early disconnects "up to the full object size". For large downloads behind flaky networks, or players that seek around video files, billed egress can be noticeably higher than delivered bytes. Free-egress providers avoid this one by design.

Units and rounding

The providers here do not agree on what a gigabyte is. AWS, Tigris, DigitalOcean and Wasabi bill binary units (GiB, or 1 TB = 1,024 GB), Storj and Hetzner decimal ones; R2 and B2 do not say. A GiB is about 7.4% larger than a decimal GB, and a TiB about 10% larger than a decimal TB. R2 rounds each line up to the next whole billing unit, and Storj rounds usage up to the whole GB. The calculator ignores all of this, which is fine for choosing a provider and not fine for forecasting a bill to the cent.

Run your own numbers

object-storage-egress-calc

$ python3 calc.py --storage-gb 500 --egress-gb 5000 --gets 20000000 --puts 10000
500 GB stored, 5,000 GB egress, 20,000,000 GETs, 10,000 PUTs, objects live 365 days (prices as of 2026-09-28)
  Wasabi (pay-as-you-go) $ 7.99 billed for 1024 GB minimum; egress above stored data: outside Wasabi's free egress policy (no overage price; service may be limited)
  Cloudflare R2 Standard $ 10.95
  Hetzner Object Storage (USD) $ 12.69
  Tigris Standard $ 19.85
  Backblaze B2 $ 38.41
  Storj Standard $ 38.50
  DigitalOcean Spaces $ 49.76
  Cloudflare R2 Infrequent Access $ 73.09
  AWS S3 Standard (us-east-1) $ 460.55

--lifetime-days applies minimum storage durations, and every price in providers.json has its source URL, so you can update a number when a provider changes it.

How to pick

If your workload is mostly	Look at first	Watch for
Downloads or media, serving more than you store	Cloudflare R2 Standard, Hetzner, Tigris	R2 read operations; Hetzner's 64 KB minimum object size
Backups and archives with little egress	Backblaze B2, Storj, Hetzner	Minimum durations if you delete early
Large public files through a CDN	B2 with a partner CDN, R2, or S3 behind a CloudFront flat-rate plan	The CDN's own terms and allowances
A small project next to your compute	DigitalOcean Spaces, R2's free tier	Where the bundle's transfer runs out
Storage you keep and rarely read, with egress below storage	Wasabi	The 1 TB minimum and 90-day minimum

What we could not include

CDN plans in front of storage, such as CloudFront's flat-rate plans or partner CDNs in front of B2. They can change the answer, but they depend on cache hit rates.
Committed-use and enterprise pricing. Contracts and volume discounts are outside this comparison.
Scaleway. Its page shows 75 GB of free egress and "EUR 0.01" after that, but does not print the unit clearly, so we left it out of the calculator rather than guess.
Performance. This is a price comparison. We did not measure latency, throughput or availability.

Summary

Zero egress is real, and for workloads that serve more than they store it changes the bill by one or two orders of magnitude compared with S3 list prices. But every provider pays for it somewhere else: a policy instead of a price, a ratio to storage, a charge per read, a minimum duration, a bundle that runs out. The right answer depends on the shape of your workload, not on the headline egress price, so run your own numbers with the fine print switched on before you move a bucket.

Originally published at devops-daily.com.

GitHub Started Enforcing Self-Hosted Runner Versions. We Tested What Happens

DevOps Daily — Wed, 30 Sep 2026 09:00:00 +0000

On September 29, GitHub started enforcing minimum versions for self-hosted Actions runners. The announcement gives one number, 2.329.0, and one rule: install each new runner release within 30 days. Neither tells you what happens to the runner you pinned in a Docker image last spring, or how long you really have before it stops. So we asked GitHub's own API about every runner release and tried to register four real runners, one version each, against a free github.com organization. The expired one did not fail loudly. It registered, connected, logged one error line and exited with code 0, and its job waited in the queue.

This post shows the data, what each runner did, and how to find every runner in your fleet before one of them goes quiet.

TLDR

We saw it on a free github.com organization. The API reports our test org's plan as free, and both the registration check and the job check fired. GitHub says Enterprise Server is not affected.
Registration floor: runner 2.328.0 was refused with "The minimum runner version required to register with GitHub Actions is now 2.329.0."
Job floor: runner 2.335.1 registered fine, then exited with "Runner version v2.335.1 is deprecated and cannot receive messages." The probe job stayed queued.
The exit looks clean: run.sh exited with code 0. With ACTIONS_RUNNER_RETURN_VERSION_DEPRECATED_EXIT_CODE=1 set, the same runner exited with code 7, which a supervisor can see.
The scheduled window: for the 18 versions with a runtime deprecation date and a later release, GitHub's API put the date 62 to 70 whole days after that next release (median 64.5). The documentation says 30.
The floor is not a safe version: 2.329.0, the registration minimum, has an API runtime deprecation date of January 23, 2026, which has passed.
On September 30, 2026, only 2.336.0 (date November 5) and 2.337.0 (no date yet) were inside their schedule, and both ran our probe job.

Prerequisites

Self-hosted runners on github.com, on VMs, bare metal, or Kubernetes with Actions Runner Controller (ARC)
The GitHub CLI (gh), logged in as an admin of at least one repository
For the fleet check: shell access to your runner hosts, or kubectl access to the cluster that runs them

What GitHub announced

The enforcement has a long history, according to GitHub's changelog. GitHub first set the 2.329.0 minimum for March 16, 2026, then paused it three days before. In June it published a new timeline: July 31 for Enterprise Cloud with data residency, September 25 for Enterprise Cloud. The last changelog post moved that to September 29.

The June post has the part that matters most, and it is easy to miss:

2.329.0 is only the registration minimum. It is the oldest runner the new Actions backend will accept.
Running jobs is a moving target. A runner has to install each new release within 30 days, or "the GitHub Actions service will stop queuing jobs to it."
Pinned runners are on their own."A runner pinned to 2.329.0 that never updates again will not pick up jobs."

In early September GitHub also shipped a REST API for runner version deprecations: GET /actions/runners/deprecations/{version} at repository, organization, or enterprise level. It is the source for the numbers below.

What GitHub's API says about every release

We listed all 146 actions/runner release records and asked the repository-level endpoint about each one. Our calls used a gh login with the classic repo scope for a user who administers the repository. The organization-level endpoint answered 403 for the same token, because it needs the admin:org scope or the fine-grained self-hosted runners permission:

gh api /repos/your-org/your-repo/actions/runners/deprecations/2.335.1


{"runner_version":"2.335.1","runtime_deprecates_at":"2026-09-24T15:30:55Z"}

Three things came back that the announcement does not say:

The API only knows 19 versions, 2.321.0 to 2.337.0. Anything older returns 404. So does 2.327.0, which was replaced by 2.327.1 three days later.
No response had a registration_deprecates_at field. The changelog lists it, but every successful response contained only runner_version and runtime_deprecates_at. The only registration rule we saw is the fixed 2.329.0 floor.
The runtime dates follow a pattern. For each version, we counted the whole days, rounded down, between the next non-prerelease release and its runtime_deprecates_at.

Days a runner version keeps running jobs after the next release ships

	Value
2.321.0	66 days
2.322.0	62 days
2.323.0	64 days
2.324.0	63 days
2.325.0	65 days
2.326.0	68 days
2.327.1	63 days
2.328.0	63 days
2.329.0	65 days
2.330.0	63 days
2.331.0	64 days
2.332.0	67 days
2.333.0	63 days
2.333.1	69 days
2.334.0	63 days
2.335.0	66 days
2.335.1	65 days
2.336.0	70 days

Documented: 30 days: 30 days

runtime_deprecates_at from GitHub's runner version deprecations API, minus the next release's publish date. 18 versions, 2.321.0 to 2.336.0. Swept 2026-09-30. GitHub's documentation says 30 days.

For every one of the 18 versions, the scheduled date fell 62 to 70 days after its successor was published. That is about nine weeks, twice the 30 days in the documentation. We do not know why. The dates may include a rollout delay, or a buffer GitHub keeps for itself.

One test lines up with the API rather than the 30 days. Thirty days after 2.337.0 was published was September 25. On September 30, a 2.336.0 runner still registered and completed our probe job, and its API date is November 5. That is one observation on one day, not a promise of extra time.

Counted from its own release date, a version's scheduled life is 66 to 138 days (median 104). Here is where recent versions stood on September 30, 2026:

Version	Released	API runtime deprecation date
2.337.0	2026-08-26	no end date yet (newest)
2.336.0	2026-07-20	2026-11-05
2.335.1	2026-06-09	2026-09-24 (ended)
2.334.0	2026-04-21	2026-08-10 (ended)
2.329.0	2025-10-14	2026-01-23 (ended)

Warning

The registration minimum is not a safe version. 2.329.0 meets the registration floor, but its API runtime deprecation date, January 23, 2026, passed eight months before enforcement started. We did not test 2.329.0 itself. The one version we tested that was past its date, 2.335.1, registered and then exited, as shown below.

What a real runner does

API dates are a schedule, not behavior. To see the behavior, we tried to register one runner per version to a private repository in a free organization, with auto-update turned off (--disableupdate) so each runner stayed on its version. Each runner that registered then got a single workflow_dispatch job. The script downloads the runner, registers it, starts it, dispatches the job, polls the run for about three minutes, and removes the runner. All output below is from runs on September 30, 2026, between 06:54 and 07:34 UTC, on a linux-arm64 host. An earlier pass of the same four tests that morning, with a first version of the script, gave the same four results; both sets of transcripts are in the repo.

Below the registration floor, 2.328.0. The runner is refused before it exists:

runner 2.328.0

$ REPO=The-DevOps-Daily/runner-floor-lab scripts/floor-test.sh 2.328.0
07:16:15Z runner 2.328.0 (arm64), label floor-2-328-0-1790752575
07:20:30Z config.sh --disableupdate
  ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ RUNNER UPDATE REQUIRED │
  ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ │
  │ The minimum runner version required to register with GitHub Actions is now 2.329.0. │
  │ Please upgrade your runner. │
  │ │
  │ For more information, see: │
  │ https://github.blog/changelog/2026-02-05-github-actions-self-hosted-runner-minimum-version-enforcement-extended │
  │ │
  └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
  Response status code does not indicate success: 404 (Not Found).
07:20:34Z config.sh exit code: 1
07:20:34Z result: not registered

This is the loud failure, and the easy one. A deploy pipeline that builds runner VMs from an old image fails at config.sh, and someone notices.

Registered, but past its end date, 2.335.1. This is the one to worry about. Registration works, the runner connects, and then:

runner 2.335.1

# the ASCII registration banner is trimmed from the output
$ REPO=The-DevOps-Daily/runner-floor-lab scripts/floor-test.sh 2.335.1
07:06:37Z runner 2.335.1 (arm64), label floor-2-335-1-1790751997
07:10:50Z config.sh --disableupdate
  # Authentication
  √ Connected to GitHub
  # Runner Registration
  √ Runner successfully added
  # Runner settings
  √ Settings Saved.
07:10:57Z config.sh exit code: 0
07:10:57Z run.sh
  runner status: offline, busy: false
07:11:34Z dispatched run 36682287765, polling for up to 180s
07:14:51Z run status after 197s: queued 
07:14:51Z runner process exited with code 0
07:14:51Z runner output (last lines):

  √ Connected to GitHub

  Current runner version: '2.335.1'
  2026-09-30 07:11:04Z: Listening for Jobs
  An error occurred: Runner version v2.335.1 is deprecated and cannot receive messages.
  Runner listener exit with terminated error, stop the service, no retry needed.
  Exiting runner...
07:15:09Z cancelled run 36682287765

Look at the order. "Runner successfully added" and "Connected to GitHub" both succeed. The runner even reports "Listening for Jobs". Then it logs one error and stops, and the process exits with code 0, the same code as a clean shutdown. The job is not rejected either. It was still queued when the script stopped polling after 197 seconds, and the script cancelled it. We did not measure how long GitHub would keep it queued.

The same runner, with the exit code turned on. Exit code 0 does not tell anything that checks exit status that the runner failed. The runner has a setting for this. With ACTIONS_RUNNER_RETURN_VERSION_DEPRECATED_EXIT_CODE=1 in its environment, run.sh exits with code 7 instead of 0 when the version is deprecated. By the runner's source and pull request #4285, the setting first shipped in 2.333.0. We tested it on 2.335.1:

runner 2.335.1, deprecated exit code on

# the ASCII registration banner is trimmed from the output
$ ACTIONS_RUNNER_RETURN_VERSION_DEPRECATED_EXIT_CODE=1 REPO=The-DevOps-Daily/runner-floor-lab scripts/floor-test.sh 2.335.1
07:25:07Z runner 2.335.1 (arm64), label floor-2-335-1-1790753107
07:29:19Z config.sh --disableupdate
  # Authentication
  √ Connected to GitHub
  # Runner Registration
  √ Runner successfully added
  # Runner settings
  √ Settings Saved.
07:29:29Z config.sh exit code: 0
07:29:29Z run.sh (ACTIONS_RUNNER_RETURN_VERSION_DEPRECATED_EXIT_CODE=1)
  runner status: offline, busy: false
07:30:04Z dispatched run 36684028728, polling for up to 180s
07:33:17Z run status after 185s: queued 
07:33:17Z runner process exited with code 7
07:33:17Z runner output (last lines):

  √ Connected to GitHub

  Current runner version: '2.335.1'
  2026-09-30 07:29:37Z: Listening for Jobs
  An error occurred: Runner version v2.335.1 is deprecated and cannot receive messages.
  Runner listener exit with deprecated version exit code: 7.
  Exiting runner...
07:33:18Z cancelled run 36684028728

Same error, and the new probe job stayed queued too, but now the process exits with a failure code. A supervisor that checks exit status can act on that. By default, systemd treats a non-zero exit as a failed service, and Kubernetes records the container's termination reason as Error instead of Completed. We did not run either of them. We set the variable in the shell that started run.sh, which is the only way we tested it.

Inside the window, 2.336.0 and 2.337.0. Both registered, picked up the job within seconds, and finished it:

runner 2.336.0

# the ASCII registration banner is trimmed from the output
$ REPO=The-DevOps-Daily/runner-floor-lab scripts/floor-test.sh 2.336.0
07:01:11Z runner 2.336.0 (arm64), label floor-2-336-0-1790751671
07:05:26Z config.sh --disableupdate
  # Authentication
  √ Connected to GitHub
  # Runner Registration
  √ Runner successfully added
  # Runner settings
  √ Settings Saved.
07:05:41Z config.sh exit code: 0
07:05:41Z run.sh
  runner status: online, busy: false
07:06:11Z dispatched run 36681802539, polling for up to 180s
07:06:29Z run status after 18s: completed success
07:06:29Z runner process: still running
07:06:29Z runner output (last lines):

  √ Connected to GitHub

  Current runner version: '2.336.0'
  2026-09-30 07:05:46Z: Listening for Jobs
  2026-09-30 07:06:14Z: Running job: probe
  2026-09-30 07:06:24Z: Job probe completed with result: Succeeded

By the API's schedule, 2.336.0 stops receiving jobs on November 5, 2026, unless it updates first.

Why this breaks quietly in practice

Runners with auto-update on should mostly follow the schedule by themselves: the runner updates when a new release is out. The risk is in setups where the runner stays on one version because it cannot or may not update:

Container images. A Dockerfile with ARG RUNNER_VERSION=2.334.0 builds the same runner every time. If those runners do not update themselves, each new one starts on the old version and would behave like the 2.335.1 runner above once its date passes.
Actions Runner Controller. ARC scale sets start runners from an image such as ghcr.io/actions/actions-runner:2.335.1, so the image tag tells you which version each new pod starts with.
VM templates and golden images. An AMI or a Packer template built in the spring holds a spring runner. A VM from it starts on that version.
--disableupdate in install scripts. Teams add it so runners do not change during a release window, then forget about them.

In our test the only clear message was in the runner's own output. The workflow run just showed "Queued".

Find every runner version you run

Start with the versions, then check them against the API. On a VM or bare-metal host, ask each installed runner directly. The runner binary prints its version:

# Run in each runner's install directory, for example /home/runner/actions-runner
./config.sh --version
# or: ./bin/Runner.Listener --version

On a fresh 2.337.0 download, both print 2.337.0. Loop over every install directory you have. A host with several runners has several versions.

On Kubernetes, list the runner images that pods run:

# Distinct runner image references, including tags such as :latest and digests
kubectl get pods -A -o jsonpath='{..image}' | tr ' ' '\n' | grep -i 'actions-runner' | sort -u

A version tag like :2.335.1 tells you the version each new pod starts with. A tag like :latest or a digest does not; resolve those before you check them. Treat both commands as a starting point, not a complete inventory.

Also check the places that build runners, not only the ones that run them: RUNNER_VERSION in Dockerfiles, Packer variables, Terraform user-data, and Helm values.

Then feed the versions to the check script. It calls the deprecations API for each version and prints the days left. It needs bash, python3 and gh, and it exits non-zero on bad input or an API error, so a broken check never reads as a pass:

check-versions

$ printf "2.337.0\nv2.336.0\n2.335.1\n2.329.0\n2.320.0\n" | REPO=your-org/your-repo scripts/check-versions.sh
VERSION ENDS STATUS
2.337.0 - no runtime deprecation date announced
2.336.0 2026-11-05 36 days left
2.335.1 2026-09-24 ended 5 days ago: stops taking jobs
2.329.0 2026-01-23 ended 249 days ago: stops taking jobs
2.320.0 - unknown to the API (it knows 2.321.0 and later): replace it

Anything that says "ended" is past its API date. In our test, the one such version we ran did not take jobs. Anything under three weeks needs a new image now.

The-DevOps-Daily/runner-version-window on GitHub

How to keep runners inside the window

Leave auto-update on for long-lived runners. It is the default. A runner that updates itself follows the schedule without anyone thinking about it. If you must control when runners change, schedule the update instead of disabling it.

Update the pinned version on every runner release. For images and templates, treat each actions/runner release as a change to make. Configure your dependency bot to bump the actual runner version or image reference, and check that it really opens those pull requests. Rebuilding an image that still pins the old version does not upgrade anything. Plan around GitHub's documented 30 days; the extra weeks we saw in the API are not a promise.

Alert on the date, not on the symptom. Run the check in CI on a schedule and fail when a version in use has less than 21 days left:

name: runner-version-check
on:
  schedule:
    - cron: '0 7 * * 1' # every Monday
  workflow_dispatch:

jobs:
  check:
    runs-on: ubuntu-latest # a GitHub-hosted runner, so the check never depends on the runners it checks
    steps:
      - uses: actions/checkout@v4
      - name: Fail if a runner version has less than 21 days left
        shell: bash
        env:
          # A token that administers REPO. The default GITHUB_TOKEN cannot call the deprecations API.
          GH_TOKEN: ${{ secrets.RUNNER_AUDIT_TOKEN }}
          REPO: ${{ github.repository }}
        run: |
          set -euo pipefail
          # Copy scripts/check-versions.sh from the companion repo into yours.
          # runner-versions.txt lists the versions your images and hosts use, one per line.
          scripts/check-versions.sh < runner-versions.txt | tee report.txt
          if grep -E 'ended|unknown|^[0-9.]+ +[0-9-]+ +([0-9]|1[0-9]|20) days left' report.txt; then
            exit 1
          elif [$? -ne 1]; then
            exit 2 # grep itself failed
          fi

Make an expired runner fail loudly. Where a container or unit starts run.sh, set ACTIONS_RUNNER_RETURN_VERSION_DEPRECATED_EXIT_CODE=1 in its environment, as in the test above, and alert on exit code 7:

# Part of a pod spec for a runner container whose command is run.sh
env:
  - name: ACTIONS_RUNNER_RETURN_VERSION_DEPRECATED_EXIT_CODE
    value: '1'

One warning, from reading the runner source, not from a test: the svc.sh service does not use run.sh. Its wrapper, bin/RunnerService.js, has no case for code 7 in 2.335.1 or 2.337.0. It treats an unknown code as a failure and starts the listener again after 5 seconds. It only gives up if GITHUB_ACTIONS_SERVICE_EXIT_AFTER_N_FAILURES is set to a positive number and that many unknown exits happen in a row. On hosts that use svc.sh, rely on the version check instead.

Watch for queued jobs. A job that sits in "Queued" for more than a few minutes on a self-hosted label is worth an alert of its own. It catches this failure and every other reason runners stop, such as capacity, networking, or a broken image.

Plan for the documented rule, not the measurement. The API scheduled every version about 64 days after its successor. The documentation says 30. Build to the documented number, and treat any extra weeks as slack you did not count on. The schedule is GitHub's to change.

What we could not test

Enterprise Cloud and data residency. We tested one free organization. The changelog names Enterprise Cloud, but we did not run a runner there.
Supervisors. We started run.sh directly. We did not test how a systemd service, the runner's svc.sh wrapper, or ARC reacts to exit code 0 or 7. The svc.sh behavior above comes from the source code.
Every version. We ran four versions. We did not test 2.329.0 itself or any other expired version besides 2.335.1.
GitHub Enterprise Server. GitHub says it is not affected, and we did not test it.
How the dates are set. The 62 to 70 day pattern comes from one snapshot of 18 versions. It is a measurement, not a published rule, and it can change.
Time. All of this is one day of data: September 30, 2026, the day after enforcement started.

Summary

The registration minimum and the runtime rule answer different questions, and GitHub's June changelog says so. 2.329.0 only decides whether a runner can register, and that failure is loud. The rule to plan around is the moving one: each version stops receiving jobs some time after the next release, 30 days by the documentation and about 64 days in the API's current schedule. In our test, the runner past its date registered, connected, logged one error and exited with code 0, while its job waited in the queue. In our direct run.sh test, one environment variable changed that to exit code 7.

Find the versions your hosts, images, and scale sets run. Check them against the deprecations API. Rebuild pinned images on every runner release, and alert on the end date before the queue tells you.

Originally published at devops-daily.com.

Terraform Muscle Memory: Practice init, plan and apply Without a Cloud Account

DevOps Daily — Tue, 29 Sep 2026 15:12:35 +0000

The awkward thing about learning Terraform is that most tutorials assume a cloud account, and a cloud account is exactly what a beginner should not point Terraform at. The gap between "read the docs" and "safely ran apply fifty times" is where the confidence comes from, and it is the gap most people never cross before their first work ticket says terraform plan.

You can cross it in a browser. Below is a practice path built on a simulated Terraform project, plus a second simulator for the part that comes after Terraform works: keeping deployed state honest over time. Disclosure: I help build these; both are free, browser-based, no signup, no AWS bill.

The loop that is the whole job

Terraform's day-to-day is one loop: change code, preview the diff, apply it, verify. The Terraform Terminal Simulator drills exactly that loop against a fake project with simulated infrastructure (you will meet a VPC with an id like vpc-04e9f2a1 that costs nobody anything). The guided sequence:

Initialize: terraform init, and what actually happens: providers download, the backend gets configured. The lesson has you run terraform validate here too, the cheap syntax-and-consistency check that belongs before every plan.
Plan: terraform plan, reading the diff before it is real. Learning to actually read a plan, not skim it, is the most transferable habit in IaC.
Apply: terraform apply, creating the simulated resources. (In real Terraform, bare apply stops for a yes/no confirmation and -auto-approve skips it; the simulator accepts both forms, and the flag is one to distrust until CI is the one running it.)
Make a change: edit, re-plan, and watch the diff show a modification instead of a creation. This is the moment Terraform's model clicks: you declare the destination, it computes the route.
Inspect state: terraform state list and terraform output, because the state file is Terraform's memory, and querying it is how you check what Terraform thinks exists.
Tear down: terraform destroy, the command that makes experiments reversible, practiced here where it can destroy nothing.

What a plan diff looks like when you make that change in step 4, the shape to learn to read:

  # aws_vpc.main will be updated in-place
  ~ resource "aws_vpc" "main" {
        id         = "vpc-04e9f2a1"
      ~ tags       = {
          ~ "Environment" = "dev" -> "staging"
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

The ~ is a modification, + is a creation, - is a destruction, and -/+ (replacement) is the one that should make you slow down: it means Terraform will destroy and recreate the resource to get there.

terraform fmt rounds out the set: it standardizes formatting so diffs stay about substance.

The reason to drill this in a sandbox first is not that the commands are hard. It is that each one has a moment where a habit forms: reading the plan instead of trusting it, checking state instead of assuming, destroying deliberately instead of clicking through. A sandbox lets those habits form before anything is at stake.

Day 2: when the repo and the cluster disagree

Terraform gets your infrastructure created. The modern follow-up question is who keeps the running system matching the repo afterwards, and that is GitOps territory. The GitOps Workflow simulator drops you into three decision scenarios, each one a real day-2 situation:

Configuration Drift: someone changed the live system by hand, so the cluster no longer matches Git. You decide how the reconciler should respond, and you see why "the repo is the truth" is a policy you enforce, not a wish.
Deployment Rollback: a bad version is live. In GitOps, the rollback is a revert in Git, not a hotfix on the cluster, and the scenario walks why that discipline pays.
Sync Policy Decision: automatic sync with self-heal and pruning, or manual gates? The scenario's preferred answer is full automation, which is worth knowing is a simplification: Argo CD itself ships with pruning and self-heal off by default, and plenty of teams gate production syncs on purpose. The value is in having to weigh it at all.

The pairing with Terraform is deliberate: plan/apply is convergence toward declared state on demand, and GitOps is the same convergence made continuous, with an agent pulling desired state and reconciling it automatically. Drift is the shared enemy.

From sandbox to real

The graduation path, in order of increasing blast radius:

The simulator until init, plan, apply, state and destroy feel boring.
Terraform against something local: HashiCorp's own getting-started path uses the Docker provider, real Terraform managing real containers on your machine, no bill possible.
A real cloud sandbox account with a budget alarm set before the first apply, not after the first bill. (Free tiers reduce cost, they do not eliminate it; the alarm is not optional.)
Reading real plans at work before you write any, because plan-reading is the skill teams actually need from a newcomer on day one.

The habits transfer unchanged; only the stakes grow.

Both simulators are part of 50+ free DevOps games and simulators. When you are ready for the real thing, HashiCorp's Docker getting-started tutorial is the natural next step: real Terraform, local resources, still no cloud bill.

containerd 1.7 Reaches End of Life: What Breaks on 2.x

DevOps Daily — Tue, 29 Sep 2026 09:00:00 +0000

containerd 1.7 reaches the end of its support window this month, and Kubernetes has already moved on: 1.35 is the last Kubernetes release that supports containerd 1.x, and 1.36 dropped it. So the question for most clusters is no longer whether to move to containerd 2, but what the move will break.

The changelog makes it sound like a lot: a new config format, removed CRI APIs, registry mirrors on their way out, schema 1 images gone. We wanted to know which of those actually bite, so we ran the upgrade. Same old node config, same data directory, containerd 1.7.36, then 2.3.6, then 2.4.1, recording everything each version said. The config still loaded. The mirrors still worked. What broke was pulling schema 1 images, even on the upgraded node. The catch is that nothing pulls on an upgraded node until something forces it, so the failure tends to appear later, on the next fresh node.

The-DevOps-Daily/containerd-2-upgrade-check on GitHub

TLDR

containerd 1.7 support ends in September 2026 , and Kubernetes dropped containerd 1.x support in 1.36.0. Recent EKS AL2023 node images already ship containerd 2.2.7.
An old version 2 config still loads on 2.x. containerd migrates it in memory at startup and logs a warning. containerd config migrate writes version = 4, and it copies the dead sections into the new file.
Inline registry mirrors still work on 2.3 and 2.4. Our logging mirror received five requests per pull on all three versions. Their removal date has moved from v2.1 to v2.4 to v2.7 depending on which version you ask.
Pulling schema 1 images is the real break. An image pulled and converted under 1.7 is still in the image store after the upgrade, but any schema 1 pull on 2.x fails: schema 1 image manifests are no longer supported. It shows up when a node has to pull: a fresh node, a garbage-collected image, or imagePullPolicy: Always.
A read-only check script in the repository reports all of this for a real node.

Prerequisites

Kubernetes nodes that run containerd. (We did not test Docker Engine hosts.)
Root access to one node, to read its config and run ctr.
kubectl if you want the cluster-wide view.

Why now

Three dates line up:

containerd's own release table lists 1.7 as an LTS release with end of life in September 2026, extended to that date for Kubernetes 1.30 to 1.32 on GKE. See RELEASES.md.
Kubernetes agreed that the last release to support containerd 1.x is 1.35, with support dropped in 1.36.0. The kubelet exposes a kubelet_cri_losing_support metric: when it appears with a version label of 1.36.0, that node's containerd is too old for the next Kubernetes version. See the Kubernetes v1.35 sneak peek.
Managed node images have moved already. The EKS AL2023 AMI release v20260923 ships containerd 2.2.7. On managed Kubernetes the containerd version usually comes with the node image, so check your node image's release notes before you upgrade a pool.

To see what every node in a cluster runs today:

kubectl get nodes -o custom-columns=NAME:.metadata.name,RUNTIME:.status.nodeInfo.containerRuntimeVersion

The experiment

The demo runs three containerd versions one after the other, against the same --root and --state directories, the way an in-place node upgrade reuses the node's data. Each one starts with the same config, written the way many Kubernetes nodes were set up in the 1.x era:

version = 2

[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.k8s.io/pause:3.9"

  [plugins."io.containerd.grpc.v1.cri".containerd]
    snapshotter = "overlayfs"
    default_runtime_name = "runc"

    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
      runtime_type = "io.containerd.runc.v2"

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
        SystemdCgroup = true

  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
    endpoint = ["http://127.0.0.1:5055"]

[plugins."io.containerd.runtime.v1.linux"]
  shim = "containerd-shim"
  runtime = "runc"

The docker.io mirror points at a tiny local server that logs every request and answers 404, so containerd falls back to Docker Hub. That turns "is this setting still honoured?" into a number we can count.

One data directory, three versions

containerd 1.7.36 old config, schema 1 pull
containerd 2.3.6 LTS, same root
containerd 2.4.1 latest, same root
runs/ every command and warning

For each version the script records ctr deprecations list, a CRI image pull through crictl, how many requests reached the mirror, the schema 1 steps, and the daemon's warnings. The terminal output below comes from those recordings; the commands are shown without the demo's socket flags, and trailing spaces are trimmed.

Finding 1: the old config still loads

Both 2.x versions started with the version 2 config. They converted it in memory and said so once, at startup:

containerd 2.3.6 daemon log

time="2026-09-28T23:41:57+03:00" level=warning msg="Configuration migrated from version 2, use `containerd config migrate` to avoid migration" t="31.333µs"

That is friendlier than the changelog suggests, and it is also a trap: a config that loads is not a config that is doing what it says. Running containerd config migrate with 2.3.6 or 2.4.1 printed a complete config with version = 4, and two details in it matter:

The runtime v1 section came through unchanged as [plugins.'io.containerd.runtime.v1.linux']. Runtime v1 was removed in 2.0, so that block now configures nothing, silently.
The inline mirrors came through too, moved to the new io.containerd.cri.v1.images section.

So config migrate gives you the new layout, not a clean config. Read the output and delete what no longer exists before you ship it.

Finding 2: the mirrors still work, and the deadline keeps moving

The inline registry.mirrors setting has been deprecated since containerd 1.5. We expected 2.x to ignore it. It did not. Each version pulled a different busybox tag through the CRI, and the logging mirror recorded five requests per pull. Here is 2.4.1:

containerd 2.4.1

$ crictl pull docker.io/library/busybox:1.38.0
time="2026-09-28T23:42:10+03:00" level=warning msg="Config \"/etc/crictl.yaml\" does not exist, trying next: \"/home/biliev/projects/containerd-2-upgrade-check/bin/crictl.yaml\""
Image is up to date for sha256:d2482869a6b838d3b4c9cad0c8085c06277909482771eb525d298fc3a7d07927
# the local mirror's request log for that pull
requests the docker.io mirror received: 5
HEAD /v2/library/busybox/manifests/1.38.0?ns=docker.io
GET /v2/library/busybox/manifests/sha256:fd7dc98638c8e305f4dc34e979f1c0fdfdcaeb0fbf8fcff77ae834b6da3d7e6e?ns=docker.io
GET /v2/library/busybox/manifests/sha256:365a051f12e05767b598e643676f14a450fb678a75ccf2beb0052c95d5c73b83?ns=docker.io

What changed is the removal date in the warning. Each version records it with ctr deprecations list:

Version	Mirrors "will be removed in"
containerd 1.7.36	v2.1
containerd 2.3.6	v2.4
containerd 2.4.1	v2.7

A deadline that has moved twice is still a deadline. There is also a side effect today: with inline mirrors in the config, 2.x logs Found 'Registry.Mirrors' in CRI config which is incompatible with transfer service ... Falling back to local image pull mode. In other words, keeping the old setting also keeps the old pull path.

The replacement is config_path plus one hosts.toml per registry:

# /etc/containerd/config.toml (version 3 or later)
[plugins.'io.containerd.cri.v1.images'.registry]
  config_path = '/etc/containerd/certs.d'


# /etc/containerd/certs.d/docker.io/hosts.toml
server = "https://registry-1.docker.io"

# Give "resolve" (tag lookups) only to mirrors you trust.
[host."https://mirror.gcr.io"]
  capabilities = ["pull"]

Finding 3: schema 1 fails, but not where you are looking

Docker image manifest schema 1 is the format images were pushed in before Docker 1.10 introduced schema 2 in 2016. Some are still served that way. scripts/find-schema1.sh asks for newer formats first, the way a current client does, and docker.io/library/busybox:1.24, gcr.io/google-containers/busybox:1.24, gcr.io/google-containers/pause:0.8.0 and quay.io/coreos/etcd:v2.2.5 still came back as schema 1. containerd 1.7 still pulls them, converting on the way and warning about it; our 1.7.36 run did that with busybox:1.24. containerd 2.0 disabled schema 1 pulls (2.0.x can turn them back on with CONTAINERD_ENABLE_DEPRECATED_PULL_SCHEMA_1_IMAGE=1), and 2.1 removed them, according to containerd's deprecation table.

On 1.7.36 the pull worked, and the deprecation was recorded:

containerd 1.7.36

# the demo keeps the last two lines of ctr's progress output
$ ctr -n demo images pull --platform linux/amd64 docker.io/library/busybox:1.24
unpacking linux/amd64 sha256:dc53f06b7ab95434d92cd20c00409fe5b5ded8d3b68f861e709f2e4b49067242...
done: 323.503075ms
$ ctr deprecations list
ID LAST OCCURRENCE MESSAGE
io.containerd.deprecation/cri-registry-mirrors 2026-09-28T20:41:42.439195342Z The `mirrors` property of `[plugins."io.containerd.grpc.v1.cri".registry]` is deprecated since containerd v1.5 and will be removed in containerd v2.1. Use `config_path` instead.
io.containerd.deprecation/pull-schema-1-image 2026-09-28T20:41:54.047880144Z Schema 1 images are deprecated since containerd v1.7 and removed in containerd v2.0. Since containerd v1.7.8, schema 1 images are identified by the "io.containerd.image/converted-docker-schema1" label.

Then we upgraded the same data directory. On 2.3.6 and on 2.4.1 the converted image was still in the image store, with its label. (It is an amd64 image and the demo ran on an arm64 Pi, so we listed it but did not run it.) Pulling it again failed:

containerd 2.4.1, same data directory

$ ctr -n demo images ls
time="2026-09-28T23:42:17+03:00" level=warning msg="DEPRECATION: The `mirrors` property of `[plugins.\"io.containerd.grpc.v1.cri\".registry]` is deprecated since containerd v1.5 and will be removed in containerd v2.7. Use `config_path` instead."
REF TYPE DIGEST SIZE PLATFORMS LABELS
docker.io/library/busybox:1.24 application/vnd.oci.image.manifest.v1+json sha256:dc53f06b7ab95434d92cd20c00409fe5b5ded8d3b68f861e709f2e4b49067242 661.3 KiB linux/amd64 io.containerd.image/converted-docker-schema1=sha256:8ea3273d79b47a8b6d018be398c17590a4b5ec604515f416c5b797db9dde3ad8
# the demo keeps the last two lines of ctr's output
$ ctr -n demo images pull --platform linux/amd64 docker.io/library/busybox:1.24
time="2026-09-28T23:42:17+03:00" level=warning msg="DEPRECATION: The `mirrors` property of `[plugins.\"io.containerd.grpc.v1.cri\".registry]` is deprecated since containerd v1.5 and will be removed in containerd v2.7. Use `config_path` instead."
ctr: schema 1 image manifests are no longer supported: invalid argument

That is the whole trap. The pull fails everywhere, the upgraded node included. But an upgraded node that already has the image on disk has no reason to pull it, so nothing visible happens on upgrade day. The failure appears later, whenever a node has to pull:

A new node from the autoscaler or a node pool upgrade has an empty image store and has to pull.
Image garbage collection on a busy node deletes the cached copy, and the next pod start pulls again.
imagePullPolicy: Always , or a rollback to an old tag that the new nodes have never pulled.

Each of these looks like a random ImagePullBackOff on one node, days after an upgrade that "went fine".

To find the images before they find you, look for the label containerd adds on conversion. The check script does this across every namespace, which on a Kubernetes node includes k8s.io. The fix is to stop depending on the old manifest: rebuild the image and push it with a current tool, which writes a schema 2 or OCI manifest, or move to a newer tag that already has one. Retagging the old image does not change its manifest.

Check a real node

scripts/check-node.sh is read-only. Run it as root on a node before you upgrade. This is its output against the demo's 1.7.36 daemon after the schema 1 pull, with the paths overridden for the demo:

check-node.sh

$ sudo CONTAINERD_CONFIG=configs/old-node-config.toml CONTAINERD_ADDRESS=/tmp/ctd-check/containerd.sock CTR=bin/containerd-1.7.36/ctr scripts/check-node.sh
== containerd version
  Version: v1.7.36
  Revision: 2892c2042ee7fbd3be0e5bdc675e07b5acedb0bf

== config: configs/old-node-config.toml
config version: 2
WARN inline registry.mirrors: deprecated; move to config_path and hosts.toml files
WARN runtime v1 section: removed in 2.0; containerd 2.x ignores it

== deprecations containerd has already seen (1.7.x and later)
ID LAST OCCURRENCE MESSAGE
io.containerd.deprecation/cri-registry-mirrors 2026-09-28T20:48:14.827082624Z The `mirrors` property of `[plugins."io.containerd.grpc.v1.cri".registry]` is deprecated since containerd v1.5 and will be removed in containerd v2.1. Use `config_path` instead.
io.containerd.deprecation/pull-schema-1-image 2026-09-28T20:48:19.606197042Z Schema 1 images are deprecated since containerd v1.7 and removed in containerd v2.0. Since containerd v1.7.8, schema 1 images are identified by the "io.containerd.image/converted-docker-schema1" label.

== images that were converted from schema 1 (they keep working; pulling them again on 2.x fails)
k8s.io docker.io/library/busybox:1.24

ctr deprecations list is the most useful line in there. It lists the deprecations containerd has observed on that node: config problems it saw at startup, and deprecated features something actually used, such as a schema 1 pull. It is not a complete audit, but it reflects the node, not the changelog.

An upgrade checklist

List runtime versions across the cluster with the kubectl command above, and watch for kubelet_cri_losing_support.
Run the check on one node per node pool or image type.
Rebuild and push every schema 1 image it finds, so it gets a schema 2 or OCI manifest, and search your manifests for old tags the check cannot see because no node has pulled them yet.
Move mirrors to config_path and hosts.toml.
Run containerd config migrate , then delete the sections for things that no longer exist, such as runtime v1, before you ship the new config.
Upgrade one node pool , then drain a node onto a fresh one, so the new node has to pull every image from scratch. That is where schema 1 fails, so test it on purpose.

What we could not test

The CRI v1alpha2 removal. containerd 2.0 removed the old CRI API. Any kubelet recent enough to still be supported uses CRI v1, but older tools that speak v1alpha2 will stop working. We had no such client to show it.
A real cluster. The recordings come from standalone daemons on a Raspberry Pi 4 (arm64). The schema 1 images are amd64 only, so we pulled them with --platform linux/amd64 and did not run them.
aufs, custom runtimes, NRI plugins and GPU setups. A node that relies on any of these needs its own test.
Every managed platform. Node images decide the containerd version and its defaults. Check your provider's release notes, as we did for EKS.

Summary

The containerd 1.7 to 2.x upgrade is gentler than its changelog. An old config loads, deprecated mirrors keep working for now, and images already on the node stay in the image store. That gentleness is exactly what makes the one real break dangerous: schema 1 pulls fail on every node, but you only see it when a node has to pull, which usually means the fresh node next week, not the one you just upgraded. Run the check, rebuild the old images, clean up the config by hand after config migrate, and test the upgrade on an empty node before the node pool does it for you.

Originally published at devops-daily.com.

Where to Run AI Agents: 8 Managed Agent Runtimes Compared

DevOps Daily — Mon, 28 Sep 2026 09:00:00 +0000

An agent that only runs while your laptop is open is a demo. The moment it has to work through a backlog overnight, react to a webhook, or run for fifty users at once, it needs somewhere else to live: a sandbox it cannot escape, state that survives a pause, a way to reach tools without holding every credential, and a bill that does not surprise you.

In 2026 that "somewhere else" became a product category. DigitalOcean put Managed Agents into public preview on 21 September, AWS shipped AgentCore Runtime V2 three days earlier, Microsoft made Foundry hosted agents generally available in July, Google renamed Vertex AI Agent Engine to Agent Runtime, and Cloudflare took Containers and its Sandbox SDK to GA in April. This post compares eight of them on the things that decide whether an agent survives production, and works out one cost scenario on every platform from list prices.

TLDR

Best overall for running coding and tool-using agents today: DigitalOcean Managed Agents. Claude Code, Codex CLI and OpenCode start with one command, each session gets its own Firecracker microVM, checkpoints capture live memory, sessions have no 8 or 24 hour limit, and 16,000+ tools sit behind one MCP endpoint. It also has the lowest vCPU list price in the group. It is a public preview in one US region, and its network is open by default until you set an allowlist.
Best for many small stateful agents: Cloudflare. Durable Objects give every agent its own storage and schedule, a hibernating agent is not billed for duration, and the Sandbox SDK adds a VM-isolated Linux box when an agent needs a shell.
Best inside an enterprise cloud: AWS AgentCore, Google Agent Runtime or Microsoft Foundry, depending on which cloud already holds your identity, network and audit trail.
Best when you run the agent loop yourself: E2B, Vercel Sandbox or Modal. They sell the sandbox, not the agent.

Prerequisites

An agent you want to host: a coding CLI such as Claude Code or Codex, a framework agent (LangGraph, ADK, Agents SDK), or your own loop.
A rough idea of your workload: session length, how much of the time the agent waits on a model, and which tools it calls.
Familiarity with the Model Context Protocol (MCP) helps for the tool-access sections.

What we compared

Every platform here promises "run your agent in the cloud". They differ on six things that matter once real work runs through them:

Isolation. Can agent-written code reach anything it should not? A microVM per session is the strongest common answer.
State. Can a session pause and resume where it was, including running processes? Can you checkpoint it and try two approaches in parallel?
Tool access. How does the agent reach GitHub, a CRM or a database, and where do the credentials live?
Time to first agent. How much packaging stands between you and a running session?
Limits. Largest sandbox, longest session, regions.
Price. List prices, and what they are billed on.

The platforms fall into four groups, and the ranking makes more sense once you see them:

Four kinds of agent runtime

Hosts for agents and coding CLIs bring a harness, get a microVM
- DigitalOcean Managed Agents
- AWS AgentCore Runtime
Framework runtimes deploy an agent built on their SDK
- Google Agent Runtime
- Microsoft Foundry hosted agents
Edge stateful agents one durable object per agent
- Cloudflare Agents SDK + Sandbox
Sandboxes for your own loop you orchestrate, they isolate
- E2B
- Vercel Sandbox
- Modal

At a glance

List prices on 28 September 2026. "Active" means CPU is billed only while it is in use.

Platform	Status	Isolation	vCPU-hour	Memory-hour	Largest sandbox	Longest session	Regions
DigitalOcean Managed Agents	Preview	Firecracker microVM	$0.044 (see note)	$0.0095 per GB, peak	16 vCPU / 32 GB	No stated limit	1 (Richmond)
Cloudflare Containers / Sandbox	GA	VM per container	$0.072, active	$0.009 per GiB	4 vCPU / 12 GiB	Not stated	Global
AWS AgentCore Runtime (V1 rates)	GA	microVM	$0.0895, active	$0.00945 per GB	2 vCPU / 8 GB	8 h (14 days on Instances)	22
Google Agent Runtime	GA	Container	$0.085	$0.009 per GiB	8 vCPU / 32 GiB	"Days"	23
Microsoft Foundry hosted agents	GA	VM per session	$0.0994	$0.0118 per GiB	2 vCPU / 4 GiB	Not stated (idle timeout 2 to 60 min)	31
E2B	GA	Firecracker microVM	$0.0504	$0.0162 per GiB	8 vCPU / 8 GiB (Hobby)	24 h (Pro)	Not stated
Vercel Sandbox	GA	Firecracker microVM	$0.128, active	$0.0212 per GB	8 vCPU (Pro), 32 (Enterprise)	24 h (Pro)	About 20
Modal Sandboxes	GA	gVisor	about $0.071	$0.024 per GiB	Not stated	24 h	Not stated

Note on DigitalOcean: its price is quoted per vCPU-hour of actual use, but active-CPU billing is "coming soon". Until then it bills 25% of the vCPUs you allocate, whatever the agent does. Microsoft's public pricing page shows placeholders; its rates above come from Microsoft's Azure Retail Prices API.

1. DigitalOcean Managed Agents

Managed Agents is two services that work together: Harness Runtime , which runs each agent session in its own Firecracker microVM, and Action Gateway , a managed MCP endpoint in front of more than 16,000 tools. It went to public preview for all users on 21 September 2026 after a private preview in August.

Why it is first. It is the only platform here where the agents most teams already use are first-class citizens. Claude Code, Codex CLI, OpenCode, Hermes and LangGraph are built-in adapters, and a custom container image covers the rest. Starting one is a single command:

doctl harness-runtime launch --harness claude-code --name repo-helper

The rest of the package is what a long-running agent needs:

Real pause and resume. Pausing "freezes the sandbox in place. Processes, memory, and the workspace filesystem are all preserved", and idle sessions pause themselves after 15 minutes by default. A paused session costs nothing for compute.
Checkpoints with live memory. A checkpoint captures the workspace and memory. From it you can fork up to four copies to try different approaches (on the Claude Code, Codex CLI and OpenCode adapters), or roll back in place. Fly.io Sprites restores files only; Daytona forks only its VM sandboxes.
Room to work. Sizes go from 1 vCPU / 1 GB up to 16 vCPU / 32 GB with a 500 GB disk, more than AWS (2 vCPU / 8 GB), Microsoft (2 / 4) or Cloudflare (4 / 12) give a single session. There is no 8 or 24 hour session limit, only a cap of 744 active hours per session per month.
Tools without credentials in the sandbox. Action Gateway exposes its catalog through three meta tools (search, invoke, and a code runner) instead of flooding the model's context. Its OAuth connections are resolved at the gateway, so the key never enters the sandbox. It also works on its own, from a Claude Code or Codex on your laptop.
One policy layer. The coding adapters share the same allow, ask and deny rules, with approvals from the terminal or with doctl harness-runtime approve. Support varies in the details: Codex CLI cannot use default: deny, and triggered runs must not use ask.
One key for every model. Instead of an Anthropic or OpenAI key, a session can use DigitalOcean serverless inference through HARNESS_INFERENCE_MODEL and HARNESS_INFERENCE_API_KEY, billed on the same DigitalOcean account.

A session is described in YAML. This one follows the environment spec reference and locks down the two defaults you most likely want to change, the open network and the permission rules:

name: repo-helper
agent: claude-code
size: mars-2vcpu-4gb
idle_timeout: 10m
env:
  HARNESS_INFERENCE_MODEL: anthropic-claude-5-sonnet
secrets:
  HARNESS_INFERENCE_API_KEY: ${DO_MODEL_ACCESS_KEY}
# Naming one host turns egress into a deny-by-default allowlist
egress:
  - api.github.com
permissions:
  default: ask
  rules:
    - tool: file.read
      action: allow
    - tool: bash
      action: ask

Price. $0.044 per vCPU-hour and $0.0095 per GB-hour of peak memory, the lowest vCPU list price in this comparison. Action Gateway calls cost $0.10 per 1,000, and the Exa search tool $10.10 per 1,000.

What to watch.

It is a public preview : no SLA, support on weekday business hours (Pacific time), no data durability guarantee, and the terms warn that the API and spec may change without notice.
It runs in one region , Richmond (RIC1), and data is processed in the US.
Egress is open by default until you name a host, and ordinary secrets are readable inside the sandbox. Scoped secrets, which keep the real key at the egress proxy, are "not yet enabled everywhere".
Billing runs from a prepaid balance shared with your other DigitalOcean products, with no per-session spend cap, and paused sessions still count toward the limit of up to 100 sessions per team.

Pick it if you want Claude Code, Codex or OpenCode working in the cloud this week, need sessions up to 16 vCPU with no fixed time limit, or want a large tool catalog without hosting MCP servers yourself.

2. Cloudflare Agents SDK, Durable Objects and Sandbox

Cloudflare approaches the problem from the other end. The Agents SDK turns each agent into a Durable Object with "a durable identity, local SQL storage, real-time connections, scheduled work, and recoverable execution", running in V8 isolates on Cloudflare's global network. When an agent needs a real Linux shell, the Sandbox SDK starts a container where "each sandbox runs in a separate VM". Containers and Sandbox went GA on 13 April 2026.

Why it is second. For agents that are many, small and long-lived (a support agent per customer, an email agent per inbox, a webhook agent per repository), it is built for exactly this. A Durable Object that hibernates is not billed for duration, CPU on Containers is billed only while active, Workers and Durable Objects run on Cloudflare's global network, and Containers can be placed by region or jurisdiction, including eu and FedRAMP. The 2026 egress controls add host allow and deny lists plus credential injection, so a sandbox can call an API without holding the key.

Price. Workers Paid ($5 a month), then Containers at $0.072 per active vCPU-hour and $0.009 per GiB-hour of provisioned memory. Durable Objects cost $0.15 per million requests and $12.50 per million GB-seconds of duration.

What to watch.

The largest container is 4 vCPU, 12 GiB and 20 GB, and container disk is ephemeral (there is a backup and restore API). Heavy coding agents outgrow it.
An agent handles 30 seconds of compute per request or message; long work goes into scheduled tasks or a sandbox.
There is no hosted tool catalog. You build and host MCP servers yourself, and McpAgent is deprecated in favour of createMcpHandler.
The agents package is still pre-1.0 (0.24.0).

Pick it if you run lots of stateful, mostly idle agents close to users, or want to host remote MCP servers at the edge.

3. AWS Bedrock AgentCore

AgentCore is AWS's set of agent building blocks: Runtime, Gateway, Identity, Memory, Code Interpreter and Browser. Each Runtime session "receives its own dedicated microVM", and Runtime V2, launched on 18 September 2026, restores sessions from snapshots with a P75 cold start of 1.9 to 2.0 seconds in AWS's own numbers.

Strengths. Runtime, Gateway and Identity run in 22 regions including GovCloud (V2 is in five so far), with IAM and VPC throughout. On V1 pricing, CPU "scales to zero during I/O wait". The Gateway turns your OpenAPI specs, Lambdas and MCP servers into tools for $0.005 per 1,000 calls. An Instances option runs sessions on EC2 capacity in your account for up to 14 days, with GPUs.

What to watch. A microVM session tops out at 2 vCPU and 8 GB and eight hours. Coding CLIs work (AWS has a walkthrough for Claude Code, Codex and others), but you package and push each one as a container yourself. The Gateway wraps your own APIs rather than shipping a SaaS catalog.

Pick it if your identity, data and audit trail already live in AWS.

4. Google Agent Runtime (formerly Vertex AI Agent Engine)

Google renamed Vertex AI Agent Engine to Agent Runtime on Gemini Enterprise Agent Platform in April 2026. It is a managed runtime for containerized agents, with full integration for Google's ADK and templates for LangGraph, LangChain, AG2 and LlamaIndex, plus Sessions and Memory Bank for state.

Strengths. $0.085 per vCPU-hour and $0.009 per GiB-hour, and "idle time spent waiting for the next prompt between turns is not billed", which suits chat-style agents that sit waiting for users. Containers go up to 8 vCPU and 32 GiB, in 23 regions, with Agent Gateway for MCP and A2A governance.

What to watch. It is a framework runtime, not a host for third-party coding CLIs. The Code Execution sandbox has "no network access" and does not let you install your own libraries.

Pick it if you build on ADK or LangGraph and already run on Google Cloud.

5. Microsoft Foundry hosted agents

Foundry Agent Service is GA, and its hosted agents reached general availability on 9 July 2026. Hosted agents use "per-session VM-isolated sandboxes" with persistent home directories, and a Toolbox puts curated tools (code interpreter, web search, OpenAPI, MCP and A2A) "behind one managed MCP-compatible endpoint".

Strengths. Entra identity, VNet integration, publishing to Teams and Microsoft 365, and 31 regions.

What to watch. Sessions are small (0.5 vCPU / 1 GiB up to 2 vCPU / 4 GiB, with up to 20 GiB of disk), idle timeouts run from 2 to 60 minutes, and hosted agents are Python and C# only. Hosted compute lists at $0.0994 per vCPU-hour and $0.0118 per GiB-hour in Microsoft's retail price API.

Pick it if your agents serve people who live in Microsoft 365.

6. E2B

E2B sells the sandbox, not the agent: "every E2B sandbox runs in its own Firecracker microVM with its own kernel", driven from its open source (Apache-2.0) SDKs while your code runs the loop.

Strengths. Pause saves "both the sandbox's filesystem and memory state", resumes in about a second, and a paused sandbox "is kept indefinitely". An MCP gateway inside the sandbox offers 200+ tools from the Docker MCP Catalog. Compute is $0.0504 per vCPU-hour and $0.0162 per GiB-hour.

What to watch. Continuous runtime is capped at 1 hour on the free Hobby plan and 24 hours on Pro ($150 a month plus usage), and compute is billed per second on the sandbox size while it runs.

Pick it if you are building a product with a code interpreter or coding agent inside it and want to own the orchestration.

7. Vercel Sandbox

Vercel Sandbox went GA in January 2026, and "each sandbox runs in its own Firecracker microVM with a dedicated kernel". Sessions can now run for 24 hours on Pro, persistence is on by default, and it is available in Vercel's compute regions.

Strengths. CPU is billed only while active ($0.128 per hour), so time spent waiting on a model is not counted, and the firewall "injects credentials into egressing traffic. The secrets never enter the sandbox." The default image ships with Node, Python and coding agents.

What to watch. No first-party MCP gateway that we found, a 45-minute session limit on Hobby, and memory at $0.0212 per GB-hour, more than twice DigitalOcean's.

Pick it if your agents are TypeScript and AI SDK code that already deploys to Vercel.

8. Modal Sandboxes

Modal runs sandboxes on gVisor, with VM sandboxes in beta, and is at its best for Python workloads and GPUs.

Strengths. Very high fan-out (Modal claims 100,000+ concurrent sandboxes), GPU support, and egress controls down to a domain allowlist. Good for RL, evals and batch agents.

What to watch. Billing is "whichever is higher: your resource request or your actual usage", sandboxes cost about three times Modal Functions, memory snapshots are alpha, and there is no first-party MCP gateway that we found.

Pick it if you run many short Python agents or need GPUs next to the sandbox.

One hour on every platform

To put the price lists side by side, take the example from DigitalOcean's launch post: one hour on a 2 vCPU / 4 GB sandbox, with the agent averaging 25% CPU. Platforms that bill active CPU pay for half a vCPU-hour; the rest pay for the full allocation. Memory is billed on whatever basis each platform uses. Free tiers, plan fees, storage, tool calls and model tokens are left out.

One hour at 2 vCPU and about 4 GB, 25% average CPU

	Value	Series
DigitalOcean	0.06$	DigitalOcean
AWS AgentCore (V1)	0.083$	Others
Cloudflare	0.108$	Others
Vercel Sandbox	0.149$	Others
E2B	0.166$	Others
Google Agent Runtime	0.206$	Others
Modal	0.238$	Others

Derived from list prices on 28 September 2026, not quotes. Cloudflare uses its predefined standard-3 type (2 vCPU / 8 GiB). Google counts the whole hour as active; its runtime does not bill idle time between turns, so a chatty agent can cost much less. Microsoft is left out because its billing basis is not confirmed.

The arithmetic for the first two, so you can redo it with your own numbers:

DigitalOcean 2 vCPU x 25% x $0.044 + 4 GB x $0.0095 = $0.022 + $0.038 = $0.060
AWS V1 0.5 vCPU-h x $0.0895 + 4 GB x $0.00945 = $0.045 + $0.038 = $0.083

Microsoft's retail rates imply about $0.246 if the full allocation is billed for the hour, but we could not confirm how hosted agents are billed.

Two things matter more than the hourly rate. First, a session that never pauses runs for 744 hours in a 31-day month: a medium DigitalOcean session costs $44.64 for that before storage, which is why auto-pause matters. Second, model tokens are billed on top of every number here. Price them before you optimise the sandbox.

How to pick

Your situation	Pick
Run Claude Code, Codex or OpenCode in the cloud with the least setup	DigitalOcean Managed Agents
Long-running sessions that need more than 8 vCPU	DigitalOcean Managed Agents
Many small, mostly idle agents close to users	Cloudflare
Everything must stay in AWS, with IAM and VPC	AWS AgentCore
ADK or LangGraph agents on Google Cloud	Google Agent Runtime
Agents for Teams and Microsoft 365 users	Microsoft Foundry
A code interpreter inside your own product	E2B
TypeScript agents already on Vercel	Vercel Sandbox
Python fan-out, evals or GPUs	Modal
Health or card data (PHI, PCI)	Not DigitalOcean while it is in preview (its terms forbid it); check each GA platform's compliance programme

What we could not confirm

Cloudflare does not name the hypervisor behind its container VMs, and Google does not name the sandboxing technology behind Agent Runtime.
Microsoft's public pricing page for hosted agents shows placeholders; the rates here come from its retail price API.
DigitalOcean's docs disagree with themselves in places: the limits page says VPC connections are not supported while the spec documents a vpc_uuid field, and resume is quoted as 200 ms in marketing and 305 ms in the launch benchmark.
Prices and preview terms change fast in this category. Check each vendor's page before you commit.

Summary

The category split into two ideas in 2026. One says "give us your agent and we will run it": DigitalOcean, AWS, Google and Microsoft. The other says "we give you a sandbox, you run the agent": E2B, Vercel, Modal, and Cloudflare with its own twist of one durable object per agent.

For most teams that want an agent like Claude Code or Codex working in the cloud now, DigitalOcean Managed Agents is the shortest path: one command to a microVM, pause and resume with memory, sandboxes up to 16 vCPU / 32 GB, a large tool catalog, and the lowest vCPU price on the list. Go in knowing it is a preview in one region, set an egress allowlist on day one, and keep an eye on the prepaid balance. If you need GA guarantees, multiple regions or a specific cloud's identity model, Cloudflare and the hyperscalers are close behind, and the decision table above tells you which.

Originally published at devops-daily.com.

Issue to Pull Request with DigitalOcean Managed Agents

DevOps Daily — Mon, 28 Sep 2026 09:00:00 +0000

We gave an agent three real GitHub issues and a microVM of its own, and kept every credential that could change the repository outside that microVM. The agent ran as OpenCode inside DigitalOcean Managed Agents, on DeepSeek V4 Pro served by DigitalOcean's own serverless inference, so the only secret it ever held was a DigitalOcean model key.

It produced three pull requests that we merged, each for about two cents of model tokens. It also produced one that passed the test suite and was still wrong, which is the most useful part of this post. Below is the whole setup, the recorded runs, the check we added after the wrong one, what the permission rules did and did not stop, and the gotchas that cost us time.

The-DevOps-Daily/do-agent-issue-to-pr on GitHub

TLDR

One script turns an issue into a pull request. It creates a Harness Runtime session, clones the repository into the microVM, prompts the agent once with no human attached, reads the diff back out, runs the tests again, and opens the pull request with credentials the agent never sees.
The model runs on DigitalOcean. HARNESS_INFERENCE_MODEL: deepseek-v4-pro plus a model access key; no Anthropic or OpenAI account involved.
Three issues, three merged fixes, about $0.02 to $0.03 of model tokens each at DigitalOcean's list price.
One pull request passed its tests and was wrong. The agent added tests where unittest never runs them. The script now refuses a change that touches tests/ without raising the number of tests that run.
Tool rules are not the boundary. OpenCode's edit and write tools were refused by our policy, so the agent wrote files with bash instead. The sandbox, the missing credentials and the egress allowlist are what actually held.

Prerequisites

A DigitalOcean account with Managed Agents (public preview since 21 September 2026) and an API token that can use it.
A DigitalOcean model access key for serverless inference.
doctl 1.175.0 or later and the GitHub CLI.
A repository with tests the agent can run. Ours is a deliberately small Python module so the runs are easy to read.

The shape of it

The target is a small duration parser, parse_duration("1h30m"), with three open issues: combined durations return the wrong number, days are not supported, and unknown units are silently ignored. All three are real bugs in the code as first committed, and the existing tests pass on all of them.

The design rule was simple: whatever the agent can touch, it cannot publish. It edits files in a microVM. Everything that talks to GitHub with write access happens outside, in a script we control.

One issue, one session, one pull request

GitHub issue labelled agent
issue-to-pr.sh holds the GitHub token
Harness Runtime OpenCode in a microVM
DO inference DeepSeek V4 Pro
Pull request a person reviews

The agent spec

A session is described in YAML. This is the file every run uses:

name: issue-to-pr
agent: opencode
size: mars-2vcpu-4gb
idle_timeout: 10m
env:
  HARNESS_INFERENCE_MODEL: deepseek-v4-pro
secrets:
  HARNESS_INFERENCE_API_KEY: ${DO_INFERENCE_KEY}
# Naming a host turns egress into a deny-by-default allowlist. The platform
# adds GitHub (for the clone) and the model endpoint on its own.
egress:
  - api.github.com
permissions:
  default: deny
  filesystem:
    mode: workspace-write
  rules:
    - tool: file.read
      action: allow
    - tool: file.write
      match: { path: '/workspace/**' }
      action: allow
    - tool: bash
      action: allow
    # Last matching rule wins, so these override the allow above.
    - tool: bash
      match: { command: 'git push*' }
      action: deny
    - tool: bash
      match: { command: 'curl *' }
      action: deny

What each part is for:

agent: opencode picks the OpenCode adapter. Managed Agents also has built-in adapters for Claude Code, Codex CLI, Hermes and LangGraph.
HARNESS_INFERENCE_MODEL and HARNESS_INFERENCE_API_KEY route the model through DigitalOcean serverless inference. The key goes under secrets, which are stored separately and never returned by the API. ${DO_INFERENCE_KEY} is filled in from your shell when you create the session.
egress is open by default. Naming a single host switches the session to an allowlist, and the platform adds the hosts the adapter needs.
permissions start from deny. For native actions the last matching rule wins, which is why the two bash denies come after the broad bash allow.

You can check a policy before paying for a session. The validate endpoint returns a verdict per rule, and all five of ours came back exact (response):

curl -X POST https://api.digitalocean.com/v2/agents/sessions/policy/validate \
  -H "Authorization: Bearer $DIGITALOCEAN_ACCESS_TOKEN" \
  -H "Content-Type: application/x-yaml" \
  --data-binary @agent.yaml

It rejects a spec that still has a ${VAR} placeholder in it, so validate a copy with the secret filled in or replaced by a dummy value.

The script, step by step

scripts/issue-to-pr.sh is about 120 lines of bash. The parts that matter:

1. A fresh microVM per issue.

doctl harness-runtime create --spec "$root/agent.yaml" --name "$session" \
  --wait-timeout 300 >/dev/null

2. Clone from outside the agent loop. The session has no GitHub credentials, so it could not clone a private repository by itself. exec runs a command in the sandbox directly, as root, so the checkout is handed to the agent user the model runs as:

doctl harness-runtime exec "$session" -- sh -c \
  "git clone -q https://github.com/$repo $workdir && chown -R agent:agent $workdir"

3. One headless run. prompt sends a single prompt, waits for the run to finish, and exits 0, 1 or 124 on timeout. --on-hitl reject refuses anything the policy would otherwise stop to ask a person about, because nobody is there to answer:

doctl harness-runtime prompt "$session" --on-hitl reject --timeout 1200 \
  -o json - <<<"$prompt" >"$out/answer.json" 2>"$out/progress.log"

The prompt includes the issue title and body and four rules: change only what the issue needs, add tests under tests/, run the test suite, and do not commit or push. With -o json the answer comes back with the run status and token counts.

4. Check the work instead of trusting the summary. The script reads the diff out of the sandbox and runs the tests there itself. It also counts the tests before and after, for a reason the next sections explain:

doctl harness-runtime exec "$session" -- sh -c \
  "git -c safe.directory=$workdir -C $workdir add -A && git -c safe.directory=$workdir -C $workdir diff --cached" \
  >"$out/change.patch"

5. Open the pull request with our credentials. The patch is applied to a fresh clone on the machine running the script, pushed to a branch named after the session, and opened with gh pr create. The agent's closing summary becomes the pull request body.

6. Remove the session. An EXIT trap saves the session's event log and removes the session, so a failed run does not leave a sandbox behind. It ignores errors from both commands, so check doctl harness-runtime list now and then.

A real run

This is issue #3, "Reject durations with unknown units", exactly as the terminal printed it:

issue-to-pr.sh

$ scripts/issue-to-pr.sh 3
12:58:31 issue #3: Reject durations with unknown units
12:58:31 creating session issue-3-1790600274
12:58:51 tests before: 8
12:58:51 running the agent
12:59:36 tests pass in the sandbox: 8 before, 10 after
remote: 
remote: Create a pull request for 'agent/issue-3-1790600274' on GitHub by visiting:        
remote: https://github.com/The-DevOps-Daily/do-agent-issue-to-pr/pull/new/agent/issue-3-1790600274        
remote: 
12:59:43 opened https://github.com/The-DevOps-Daily/do-agent-issue-to-pr/pull/7
https://github.com/The-DevOps-Daily/do-agent-issue-to-pr/issues/3#issuecomment-5870355576
12:59:49 removing session issue-3-1790600274

The change it produced, pull request #7, checks the whole string before parsing it:

 _UNITS = {"s": 1, "m": 60, "h": 3600, "d": 86400}
 _PART = re.compile(r"(\d+)([smhd])")
+_VALID = re.compile(r"(\d+[smhd])+")

 def parse_duration(text: str) -> int:
     text = text.strip().lower()
     if not text:
         raise ValueError("empty duration")
+ if not _VALID.fullmatch(text):
+ raise ValueError(f"not a duration: {text!r}")

It also added two tests, one for "1h30x" and one for "abc". A reviewer would point out that the old "no parts" check below it is now unreachable, which is the kind of note that belongs in a review, not a reason to reject the fix. We merged it. Issue #1 went the same way: a one-character fix (total = to total +=) and two tests, merged as #4.

The pull request that passed and was wrong

Issue #2 asked for days, "7d" and "1d12h". The first run changed the code correctly and reported that all tests passed, and our script, which at that point only checked that the suite passed, opened pull request #5.

The tests it added looked like this at the bottom of the file:

if __name__ == " __main__":
    unittest.main()

    def test_days(self):
        self.assertEqual(parse_duration("7d"), 604800)

They are inside the if __name__ block, not inside the test class, so unittest never collects them. The agent's own summary said "All 8 tests pass (6 pre-existing + 2 new ones)". The test run our script did afterwards in the same sandbox printed Ran 6 tests.

A person reading the diff caught it. The script could not, because "the tests pass" was the only thing it checked. So it now counts how many tests run before and after the agent's change, and refuses to open a pull request if the agent touched tests/ but the count did not go up. It is a coarse check, since it cannot tell which new test ran, but it catches this failure:

count_tests() {
  doctl harness-runtime exec "$session" -- sh -c \
    "cd $workdir && python3 -m unittest -q 2>&1" | sed -n 's/^Ran \([0-9]*\) tests\{0,1\}.*/\1/p'
}

The second run of issue #2 added four tests that ran (6 before, 10 after), but failed to push because pull request #5's branch still existed; each run now pushes to its own branch. The third run added two tests that ran and became #6, which we merged.

The lesson is not specific to this platform. An agent's report of its own work is a claim. Check the claim with something the agent did not write, and keep a person on the merge button.

What the policy stopped, and what it did not

The network allowlist held. With egress limited to api.github.com, we ran these inside a session with doctl harness-runtime exec (recorded here):

inside the session

$ python3 -c "import urllib.request as u; print(u.urlopen('https://example.com', timeout=8).status)"
urllib.error.URLError: <urlopen error Tunnel connection failed: 404 Not Found>
$ python3 -c "import urllib.request as u; print(u.urlopen('https://pypi.org/simple/', timeout=8).status)"
urllib.error.URLError: <urlopen error Tunnel connection failed: 404 Not Found>
$ python3 -c "import urllib.request as u; print(u.urlopen('https://github.com', timeout=8).status)"
200

GitHub and the model endpoint still work, because the platform adds those hosts itself. An agent talked into leaking something by a malicious issue cannot reach an arbitrary server, though it can still reach the hosts on the list.

The tool rules did not do what we expected. The session log for issue #3 shows OpenCode's own editing tools being refused:

doctl harness-runtime logs issue-3-1790600274

▸ edit
[2026-09-28T12:58:59Z] run.tool_call_completed
  ✗ The user has specified a rule which prevents you from using this specific tool … (22ms)

The file.write rule for /workspace/** did not cover OpenCode's edit and write tools in these runs, so default: deny refused them. The agent did not stop. It changed the files through bash instead, with sed -i and cat > heredocs, which we had allowed. Every merged fix in this post was written that way.

DigitalOcean's docs warn about exactly this: "Rules match literally ... A denied action doesn't block the goal behind it." If bash is allowed, the agent can do anything bash can do inside the sandbox, whatever the other rules say. Treat tool rules as a way to shape the agent's behaviour, and treat the microVM, the missing credentials and the egress allowlist as the security boundary.

What it cost

Token counts come from the prompt command's JSON output; the price is DigitalOcean's list price for DeepSeek V4 Pro, $1.74 per million input tokens and $3.48 per million output tokens.

Run	Outcome	Input / output tokens	Model cost
Issue #1	Merged (#4)	8,481 / 1,024	$0.018
Issue #2, first	Closed in review (#5)	14,372 / 1,848	$0.031
Issue #2, second	Refused to push (old branch)	14,676 / 2,268	$0.033
Issue #2, third	Merged (#6)	9,868 / 1,605	$0.023
Issue #3	Merged (#7)	8,750 / 2,109	$0.023

Two smaller costs sit on top. Each new session starts with a short readiness run, which used between 235 and 4,587 input tokens in ours. And the sandbox is billed while it exists: a Medium session lists at about $0.060 an hour under the current billing rule (25% of the allocated vCPUs plus peak memory), and by the script's timestamps each of ours existed for under two minutes, so compute came to a fraction of a cent per issue. These figures are derived from token counts and list prices; the new usage had not reached the invoice when we wrote this.

Gotchas we hit

doctl wants an Anthropic key for Claude Code, even on DigitalOcean inference. doctl harness-runtime create refused a claude-code spec that used HARNESS_INFERENCE_*, asking for ANTHROPIC_API_KEY. In doctl's source, prepareClaudeCodeStart resolves that key for every claude-code session and checks it against Anthropic. Posting the same YAML to POST /v2/agents/sessions with Content-Type: application/x-yaml created the session (both recorded). OpenCode has no such check.
Model access depends on your inference tier. Our key got "403 this model is not available for your subscription tier" for the Anthropic and OpenAI models we tried, both in a direct chat completion and inside a Claude Code session. Open models such as DeepSeek V4 Pro worked. Test the model you want with a plain chat completion before building on it.
--gh-repo did not clone anything for us. Without a GitHub connection set up through doctl harness-runtime auth, the workspace stayed empty. Cloning with exec is explicit and works for public repositories.
exec runs as root. Files it creates belong to root, and the agent could not edit them until we added chown. Git run as root then refuses the agent-owned checkout as "dubious ownership" unless you pass -c safe.directory.
A closed pull request leaves its branch. Name branches after the session, not the issue, or the next attempt cannot push.

Wiring it to GitHub Actions

The repository includes a workflow that runs the same script when an issue gets the agent label:

on:
  issues:
    types: [labeled]

jobs:
  agent:
    if: github.event.label.name == 'agent'
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
      - uses: digitalocean/action-doctl@v2
        with:
          token: ${{ secrets.DIGITALOCEAN_ACCESS_TOKEN }}
      - run: scripts/issue-to-pr.sh "${{ github.event.issue.number }}"
        env:
          GH_TOKEN: ${{ github.token }}
          DIGITALOCEAN_ACCESS_TOKEN: ${{ secrets.DIGITALOCEAN_ACCESS_TOKEN }}
          DO_INFERENCE_KEY: ${{ secrets.DO_INFERENCE_KEY }}

Two things to know before you turn it on. First, the DigitalOcean token in that secret should be able to do as little as possible, ideally in a separate DigitalOcean team: a token that can manage your whole account is a large thing to hand to a workflow. Second, pull requests opened with the workflow's own GITHUB_TOKEN do not trigger other workflows, so your normal test workflow will not run on them unless you open them with a GitHub App or a separate token. The script's own test run inside the sandbox covers the gap, but it is not a substitute for CI.

The runs in this post were made with the script from a terminal, not through Actions.

What we could not conclude

Whether this scales past a toy repository. The target is a few dozen lines with a fast test suite. A real codebase means longer runs, more tokens, and tests the agent may not be able to run inside the sandbox without more egress.
Why the file.write rule did not cover OpenCode's edit tools. We saw the refusals, not the mapping behind them.
How other models compare. Our inference tier only allowed open models, so we did not run the same issues through Claude or GPT.
Anything about speed. This is a how-to, not a benchmark, and Managed Agents is a public preview.

Summary

The pattern is small and reusable: a disposable microVM per task, a model key as the only secret inside, an egress allowlist, a script outside the sandbox that checks the agent's work with something the agent did not write, and a person on the merge button. On DigitalOcean Managed Agents the whole loop is a create, a few exec calls, one prompt and a remove, and the model can run on DigitalOcean too.

The agent did good work on three small issues for a few cents each. It also produced a confident pull request whose new tests never ran, and it worked around refused editing tools through bash without being asked to. Both are reasons to build the checks first and the automation second.

Originally published at devops-daily.com.

Terraform State: Remove, Move and Migrate Resources, and Set Up a Remote Backend

DevOps Daily — Sat, 26 Sep 2026 09:00:00 +0000

Terraform state is the file that maps every resource in your configuration to a real object somewhere: an instance ID, a bucket name, a DNS record. Most days you never look at it. Then someone renames a resource, splits a repository, or needs Terraform to let go of a database without deleting it, and suddenly the state file is the only thing that matters.

This post covers the state operations that come up again and again: removing a resource from state, renaming or moving it, carrying it to another project, bootstrapping a remote backend, and the DynamoDB error people hit while building a lock table. Each one has an old way (a CLI command that edits state directly) and, in recent Terraform versions, a new way (a block in your configuration that goes through plan like any other change). The new way is almost always better, and the sections below show why.

All the terminal output in this post comes from real runs on Terraform 1.15.8, using the built-in terraform_data resource so the examples run anywhere without a cloud account.

TLDR

To stop managing a resource without destroying it, use a removed block with destroy = false (Terraform 1.7+). terraform state rm does the same thing but skips plan, and it will recreate the resource if you forget to delete it from the config.
To rename a resource, use a moved block (Terraform 1.1+). It shows up in plan and gets code review. terraform state mv still works for one-off fixes.
To move a resource to another project, remove it from the old one with a removed block and adopt it in the new one with an import block. Editing state files by hand is the fallback.
To bootstrap a remote backend, create the bucket with local state first, then add the backend block and run terraform init -migrate-state.
The S3 backend can lock with a lock file in the bucket (use_lockfile = true, added in Terraform 1.10). DynamoDB locking has been deprecated since 1.11. Lock files also work on S3-compatible storage like DigitalOcean Spaces.
Never commit .tfstate to Git. Do commit .terraform.lock.hcl.

Prerequisites

Terraform 1.7 or later for removed blocks, and 1.11 or later for the S3 lock-file examples (the feature arrived in 1.10 as experimental)
A configuration with existing state to practice on, or the Terraform terminal simulator if you want to try terraform state list in the browser first
Access to an object storage bucket (AWS S3 or DigitalOcean Spaces) for the backend sections

What state actually tracks

Configuration what you want
State what Terraform thinks exists
Real infrastructure what actually exists

By default, every plan compares three things: your configuration, the state file, and the real objects the provider can see. State is the link in the middle. It is keyed by resource address (aws_instance.web, module.network.aws_vpc.main), so almost every problem in this post comes down to one of two questions: which address points at which real object, and which state file holds that address.

You can see the addresses in your state at any time:

terraform state list

$ terraform state list
terraform_data.db
terraform_data.web

Stop managing a resource without deleting it

The situation: a database, a DNS zone or a bucket was created by Terraform, and now it should live outside this configuration. Maybe another team owns it, maybe you are splitting a repository. You want Terraform to forget it, not destroy it.

The old way: terraform state rm

state rm, then plan

$ terraform state rm terraform_data.db
Removed terraform_data.db
Successfully removed 1 resource instance(s).
$ terraform plan
...
Plan: 1 to add, 0 to change, 0 to destroy.

That second command is the trap. state rm removed the resource from state, but the resource "terraform_data" "db" block is still in the configuration, so the next plan wants to create it again. On a real database that means a second database, or a name collision. state rm only works safely when you delete the block from the configuration in the same change, and nothing in the workflow reminds you to do that.

It also skips plan entirely. The change happens the moment you press enter, it never shows up in a pull request, and in CI there is nothing to review.

The new way: a removed block

Delete the resource block and put a removed block in its place:

removed {
  from = terraform_data.db

  lifecycle {
    destroy = false # forget it, do not delete it
  }
}

Now the removal is a normal change that goes through plan:

plan with a removed block

$ terraform plan
Terraform will perform the following actions:
  # terraform_data.db will no longer be managed by Terraform, but will not be destroyed
  # (destroy = false is set in the configuration)
  . resource "terraform_data" "db" {
        id = "a38fe64a-ecd1-489f-f733-8048076db5f0"
        # (2 unchanged attributes hidden)
    }
Plan: 0 to add, 0 to change, 0 to destroy.

Warning: Some objects will no longer be managed by Terraform
$ terraform apply -auto-approve
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
$ terraform state list
terraform_data.web

The plan says exactly what will happen, your reviewer sees it, and there is no window where the configuration and the state disagree. Once the change is applied everywhere, you can delete the removed block. If you leave destroy out (it defaults to true), the block becomes a way to destroy a resource on purpose, which is also useful, just not here.

Tip

Before any state change, take a copy: terraform state pull > backup.tfstate. It costs nothing and gives you a way back. Restoring it after another change is not a plain push: in our run terraform state push backup.tfstate refused with "cannot import state with serial 3 over newer state with serial 4". terraform state push -force backup.tfstate overwrites the current state and skips both the serial and the lineage checks, so treat it as an exception: take a fresh backup of the current state first, and remember it restores Terraform's records, not the infrastructure.

Rename or move a resource inside a project

Renaming aws_instance.web to aws_instance.frontend looks harmless in the code. Terraform sees it differently: one address disappeared and a new one appeared, so the plan destroys the old instance and creates a new one. For anything with data on it, that is an outage.

The new way first: a moved block

resource "terraform_data" "frontend" {
  input = "web-server"
}

moved {
  from = terraform_data.web
  to = terraform_data.frontend
}

plan with a moved block

$ terraform plan
Terraform will perform the following actions:
  # terraform_data.web has moved to terraform_data.frontend
    resource "terraform_data" "frontend" {
        id = "0f0915bf-cbea-ff5f-1129-a346d2268e84"
        # (2 unchanged attributes hidden)
    }
Plan: 0 to add, 0 to change, 0 to destroy.

No destroy, no create, just a move that anyone can read in the pull request. moved blocks also handle the moves that are painful by hand: pulling resources into a module (from = aws_s3_bucket.logs, to = module.logging.aws_s3_bucket.this), or switching from count to for_each (from = aws_instance.web[0], to = aws_instance.web["primary"]).

A moved block is cheap to keep. In a reusable module, keep it for good: consumers can skip versions, and a removed moved block turns their upgrade into a destroy and recreate. In a root configuration you can delete it once every state that uses the configuration has applied the move.

The old way: terraform state mv

terraform state mv

$ terraform state mv terraform_data.frontend terraform_data.web
Move "terraform_data.frontend" to "terraform_data.web"
Successfully moved 1 object(s).
$ terraform plan
No changes. Your infrastructure matches the configuration.

Here we undid the rename from the previous section: after applying the moved block, we renamed the block back to web, deleted the moved block, and moved the state to match. It works, and for a quick fix on your own sandbox it is fine. The problem is the same as state rm: it changes shared state immediately, outside review, and the code change that goes with it has to land separately. In a team, prefer moved. Both state rm and state mv accept -dry-run if you want to see what they would touch first.

Move resources to another project

Splitting a large configuration into smaller ones (network in one project, applications in another) means carrying resources from one state file to a different one. There are two ways to do it.

The reviewable way: removed plus import

In the source project, delete the resource and let go of it:

removed {
  from = aws_instance.web

  lifecycle {
    destroy = false
  }
}

In the target project, add the resource block and adopt the existing object with an import block (Terraform 1.5+):

import {
  to = aws_instance.web
  id = "i-0a1b2c3d4e5f67890" # the real instance ID
}

resource "aws_instance" "web" {
  ami = "ami-0c55b159cbfafe1f0"
  instance_type = "t3.small"
}

Apply the source first, so the object is only ever managed by one project, then the target. Make sure nobody runs either project in between. The target's plan shows 1 to import, and if your resource block does not match the real object, the plan shows the differences before anything changes. If you would rather not write the resource block by hand, leave it out, keep only the import block, and run terraform plan -generate-config-out=generated.tf (to a file that does not exist yet): Terraform writes a starting block from the real object. Both changes go through plan and review, and at no point does anyone edit a state file.

The fallback: move between state files directly

Some resources cannot be imported, and sometimes you need to move dozens at once. terraform state mv can write to a different state file. It only moves state, so move the configuration in the same change: delete the resource block from the source, add it to the target, and fix any references. With a remote backend, pull both states to local files, move the resource, and push them back:

# in the source project
terraform state pull > source.tfstate

# in the target project
terraform state pull > target.tfstate
terraform state mv -state=../source/source.tfstate -state-out=target.tfstate \
  aws_instance.web aws_instance.web
terraform state push target.tfstate

# back in the source project
terraform state push source.tfstate

Here is the core of it on two local projects:

move a resource to another project

$ terraform state mv -state=../source.tfstate -state-out=../network/terraform.tfstate terraform_data.web terraform_data.web
Move "terraform_data.web" to "terraform_data.web"
Successfully moved 1 object(s).
# in the target project, which now holds the resource
$ terraform state list
terraform_data.web
$ terraform plan
No changes. Your infrastructure matches the configuration.

In this run the target configuration already contained the terraform_data.web block, which is why its plan shows no changes. Nobody else should run Terraform against either project while you do this, and you should keep the backups from the tip above. Like state rm and state mv inside one project, it changes state without a plan, so check both plans right after.

Set up a remote backend, with Terraform itself

State in a local terraform.tfstate file works for one person on one laptop. The moment a second person or a CI job runs Terraform, you need a remote backend with locking turned on for every writer, so two applies cannot write the same state at once. Not every backend locks, and on S3 it is opt-in, so check that yours does.

The classic chicken-and-egg problem: you want Terraform to create the bucket that will hold Terraform's state. The answer is two steps.

Step 1: create the bucket with local state. A small bootstrap configuration, applied once:

Bootstrap the state bucket

AWS S3

resource "aws_s3_bucket" "state" {
  bucket = "acme-terraform-state"
}

resource "aws_s3_bucket_versioning" "state" {
  bucket = aws_s3_bucket.state.id
  versioning_configuration {
    status = "Enabled" # every state write becomes a recoverable version
  }
}

resource "aws_s3_bucket_public_access_block" "state" {
  bucket = aws_s3_bucket.state.id
  block_public_acls = true
  block_public_policy = true
  ignore_public_acls = true
  restrict_public_buckets = true
}

DigitalOcean Spaces

resource "digitalocean_spaces_bucket" "state" {
  name = "acme-terraform-state"
  region = "fra1"
  acl = "private"
}

Step 2: point the configuration at the bucket and migrate. Add a backend block:

Backend block

AWS S3

terraform {
  required_version = ">= 1.11"

  backend "s3" {
    bucket = "acme-terraform-state"
    key = "prod/network.tfstate"
    region = "us-east-1"
    encrypt = true
    use_lockfile = true # lock with a .tflock object in the bucket, no DynamoDB
  }
}

DigitalOcean Spaces

terraform {
  required_version = ">= 1.11"

  backend "s3" {
    endpoints = {
      s3 = "https://fra1.digitaloceanspaces.com"
    }
    bucket = "acme-terraform-state"
    key = "prod/network.tfstate"

    # Spaces speaks the S3 API; these skip AWS-only checks
    skip_credentials_validation = true
    skip_requesting_account_id = true
    skip_metadata_api_check = true
    skip_region_validation = true
    skip_s3_checksum = true
    region = "us-east-1" # required by the backend, not used by Spaces

    use_lockfile = true
  }
}

Then run terraform init -migrate-state. Terraform notices the backend changed, asks whether to copy the existing state to the new backend, and from then on reads and writes it remotely. The same command handles any backend change later: a new bucket, a new key, or moving from one provider to another. Here it is moving local state to a new location:

terraform init -migrate-state

$ terraform init -migrate-state -force-copy
Initializing the backend...
Successfully configured the backend "local"! Terraform will automatically
use this backend unless the backend configuration changes.
...
Terraform has been successfully initialized!

-force-copy answers yes to the copy prompt, which is what you want in a script. Run it without the flag the first time so you see the question.

For Spaces, the credentials are a Spaces access key and secret, passed as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the environment, not written in the backend block. DigitalOcean documents the full setup in Configure DigitalOcean Spaces as a Terraform Remote State Backend, including state locking with use_lockfile on Terraform 1.11 or later. If your state also holds secrets (it usually does), keep the bucket private and limit who has keys to it; that is the whole point of the next two sections.

For the errors people usually hit on this step, from the wrong region to missing permissions, see common S3 backend configuration errors, and if a lock gets stuck, how to unlock a locked state file.

The DynamoDB lock table is on its way out

For years, locking on the S3 backend meant a separate DynamoDB table with a LockID key. Terraform 1.10 added locking through S3 itself: at the start of an operation Terraform writes a .tflock object next to the state with a conditional write that fails if the object already exists, so a second apply is blocked. With use_lockfile = true you no longer need the table, and Terraform now says so when it sees the old setting:

terraform init with dynamodb_table

$ terraform init
Initializing the backend...
Warning: Deprecated Parameter
  on main.tf line 6, in terraform:
   6: dynamodb_table = "terraform-locks"
The parameter "dynamodb_table" is deprecated. Use parameter "use_lockfile"
instead.

If you are migrating an existing backend, you can set both use_lockfile = true and dynamodb_table for a while. A client with both settings takes both locks, so it excludes old clients that only know DynamoDB and new ones that only use the lock file. Retire the table only when every writer (people and CI jobs) has use_lockfile enabled, has permission to create and delete the .tflock object, and nothing depends on DynamoDB any more.

The "all attributes must be indexed" error

If you still build a DynamoDB table in Terraform, for locking or anything else, you will probably meet this error from the AWS provider sooner or later:

Error: all attributes must be indexed. Unused attributes: ["category"]

The message sounds like a query rule, but it is about the attribute blocks. In aws_dynamodb_table, an attribute block does not describe the item's fields. It declares the type of a key : the table's hash_key or range_key, or a key of a global or local secondary index. DynamoDB is schemaless for every other field, so an attribute that no key uses is an error.

The wrong way, declaring fields as if it were a SQL table:

resource "aws_dynamodb_table" "orders" {
  name = "orders"
  billing_mode = "PAY_PER_REQUEST"
  hash_key = "id"

  attribute {
    name = "id"
    type = "S"
  }

  attribute {
    name = "category" # no key uses this: "all attributes must be indexed"
    type = "S"
  }
}

The right way: declare only key attributes. If you do need to query by category, make it a key of an index, and then its attribute block is valid:

resource "aws_dynamodb_table" "orders" {
  name = "orders"
  billing_mode = "PAY_PER_REQUEST"
  hash_key = "id"

  attribute {
    name = "id"
    type = "S"
  }

  attribute {
    name = "category"
    type = "S"
  }

  global_secondary_index {
    name = "by-category"
    hash_key = "category"
    projection_type = "ALL"
  }
}

For a lock table, the rule is short: one attribute, LockID of type S, as the hash key, and nothing else.

Should .tfstate go in Git?

No, for three reasons that each matter on their own:

State can hold secrets in plain text. Database passwords, generated keys, and values marked sensitive are redacted in plan output, but sensitive does not keep them out of state. (Newer ephemeral values and write-only arguments do, where a provider supports them.) A state file in Git is a credential in Git, forever, in every clone and every fork.
Git cannot lock. Two people who run apply from their own checkouts each write a different state. The next merge picks one, and Terraform loses track of whatever the other one created.
State changes whenever an apply changes something. Committing it makes every infrastructure change a merge conflict waiting to happen.

What belongs where:

# .gitignore
*.tfstate
*.tfstate.*
.terraform/
crash.log
# saved plans can contain sensitive values: save them as *.tfplan
*.tfplan
# only if your variable files hold secrets; commit a non-secret example instead
*.tfvars
*.tfvars.json

Commit .terraform.lock.hcl, though. It pins the exact provider versions and checksums, so everyone and every CI run use the same provider build. And if state already made it into your history, rotating the secrets in it matters more than rewriting the history, because every existing clone still has the old file.

Where state should live is a bigger question than a backend block: who is allowed to run apply, and whose laptop has the keys. We wrote about that in Who owns the state file.

Summary

You want to	Use	Instead of
Stop managing a resource, keep it running	`removed` with `destroy = false`	`terraform state rm`
Rename a resource or move it into a module	`moved` block	`terraform state mv`
Move a resource to another project	`removed` in the source, `import` in the target	pulling, editing and pushing state files
Start using a remote backend	bootstrap the bucket, then `terraform init -migrate-state`	copying state files by hand
Lock state on S3 or Spaces	`use_lockfile = true`	a DynamoDB table
Keep state safe	a private, versioned bucket	committing `.tfstate` to Git

The pattern behind all of it: prefer changes that go through plan. The config blocks (removed, moved, import) turn state surgery into reviewable code, and the CLI commands are there for the rare case where that is not possible. If you want to go further with the language itself, Terraform variables, loops and outputs covers the rest, and the Terraform terminal simulator lets you practice init, plan, apply and state list in the browser.

Originally published at devops-daily.com.

Terraform Module Outputs, Inputs and Sources: How Modules Pass Data to Each Other

DevOps Daily — Sat, 26 Sep 2026 09:00:00 +0000

A Terraform module is a box with a very small door. Everything inside it (resources, locals, data sources) is private. The only way in is an input variable, and the only way out is an output. Once that clicks, most module questions answer themselves: "how do I reference the instance my module created?", "how do I give this module my VPC?", "why can't I see module.app.aws_instance.web?"

This post walks through the module questions that come up most in real projects: outputs, passing values and whole resources between modules, modules with for_each, Git sources pinned to a branch or tag, and the provider error people hit when they refactor old modules. Every command in the terminal blocks was run on Terraform 1.15.8 with the built-in terraform_data resource and the random provider, so you can repeat them without a cloud account.

TLDR

A module exposes values only through output blocks. From the caller you read them as module.<name>.<output>. You cannot reach into a module's resources directly.
To pass data into a module, declare a variable in the module and set it in the module block. That is how you "pass a resource" too: pass its attributes, or the whole object with a typed variable.
Referencing module.network.vpc_id from another module creates the dependency automatically. No depends_on needed.
A module with for_each becomes a map of instances: module.network["production"].vpc_id, or a for expression to collect them all.
Git module sources take ?ref= with a branch, tag or commit. Use tags or commits for anything shared, branches only while developing.
"Provider configuration not present" means Terraform needs a provider configuration that is gone, most often because a module with its own provider block was removed while its resources are still in state. Move provider blocks to the root, apply, then remove the module.

Prerequisites

Terraform 1.4 or later for the examples (they use the built-in terraform_data resource; we ran them on 1.15), and 1.7 or later for the removed block mentioned at the end
A root configuration with at least one local module, or the Terraform terminal simulator to practice the basics first
For the Git source section, access to a Git repository that holds a module

The layout used in this post

.
├── main.tf # root module: calls the child modules
└── modules
    ├── network
    │ └── main.tf # creates a "VPC", outputs its id and CIDR
    └── app
        └── main.tf # takes a vpc_id input, creates a "server"

Data only crosses a module boundary through variables and outputs

Root module main.tf
module.network for_each: staging, production
module.app var.vpc_id
Root outputs vpc_ids, app_server

Connections:

Root module -> module.network (cidr, name)
module.network -> module.app (vpc_id output)
module.network -> Root outputs (vpc_id)
module.app -> Root outputs (server_id)

Get a value out of a module: outputs

Inside the module, an output block decides what the caller can see:

# modules/network/main.tf
variable "name" { type = string }
variable "cidr" { type = string }

resource "terraform_data" "vpc" {
  input = { name = var.name, cidr = var.cidr }
}

output "vpc_id" {
  description = "ID of the network, for modules that need to attach to it"
  value = terraform_data.vpc.id
}

output "cidr" {
  value = var.cidr
}

In the caller, the module's outputs are attributes of module.<name>. That is the only view you get. Try to reach past it, to the resource itself, and Terraform refuses:

reaching inside a module

# an output that tries to read a resource inside module.app
$ terraform plan
Error: Unsupported attribute
  on extra.tf line 2, in output "direct":
   2: value = module.app.terraform_data.server.id
    ├────────────────
    │ module.app is object with 1 attribute "server_id"
This object does not have an attribute named "terraform_data".

The error message is the whole lesson: module.app is object with 1 attribute "server_id". If the caller needs something, the module has to output it.

A few habits make outputs easier to live with:

Output IDs and names, not whole resources, by default. Callers then depend on a small, stable interface instead of every attribute of a resource type.
Add description. It shows up in module documentation generators and in the registry.
Mark secrets sensitive = true. Terraform then hides the value in plan and apply output. The value is still stored in state, so that does not make it safe to share state.
You can output a whole object when the caller genuinely needs many attributes: value = aws_instance.web gives the caller module.app.web.private_ip, module.app.web.arn and so on, at the cost of a wider interface.

Pass a resource into a module: inputs

There is no way to hand a module "the resource" as a live link. You pass values through variables, and Terraform tracks the dependency for you.

The common case is a single attribute:

# modules/app/main.tf
variable "vpc_id" {
  description = "Network the server attaches to"
  type = string
}

resource "terraform_data" "server" {
  input = "server in ${var.vpc_id}"
}

output "server_id" {
  value = terraform_data.server.id
}


# main.tf
module "app" {
  source = "./modules/app"
  vpc_id = module.network["production"].vpc_id # an output of another module
}

When a module needs several attributes of the same resource, pass them together as an object instead of five separate variables:

variable "network" {
  type = object({
    id = string
    cidr = string
  })
}


module "app" {
  source = "./modules/app"
  network = {
    id = module.network["production"].vpc_id
    cidr = module.network["production"].cidr
  }
}

You can even pass a whole resource object into a variable, as long as the object type matches its attribute names. An aws_vpc has id and cidr_block, so the variable would be object({ id = string, cidr_block = string }) and the caller writes network = aws_vpc.main; Terraform keeps the attributes the type names and discards the rest. type = any also works. The typed object is better: it documents exactly which attributes the module relies on, and the error messages are clearer when something does not fit.

Chain modules: one module's output as another's input

The module "app" block above already chains two modules. Because it reads module.network["production"].vpc_id, Terraform knows that everything in module.app that uses the value depends on the network. You do not need depends_on. Reach for depends_on on a module only for dependencies Terraform cannot see from references, and even then prefer passing a real value: depends_on on a module makes every resource in it wait for everything it depends on, which can make plans noisier.

The pattern scales to longer chains, network to database to application, with each module taking the previous one's outputs as inputs. Keep the chain in the root module. A child module that calls another module to get at a third one's outputs is usually a sign that the boundaries are wrong.

Modules with for_each: outputs become a map

Put for_each on a module block and you get one instance per key:

module "network" {
  source = "./modules/network"
  for_each = {
    staging = "10.10.0.0/16"
    production = "10.20.0.0/16"
  }
  name = each.key
  cidr = each.value
}

output "vpc_ids" {
  value = { for env, net in module.network : env => net.vpc_id }
}

output "app_server" {
  value = module.app.server_id
}

module.network is now a map of module instances. Index it with a key to read one (module.network["production"].vpc_id), or loop over it with a for expression to collect an output from every instance. Here is the whole configuration applied:

module outputs with for_each

$ terraform apply -auto-approve
...
Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Outputs:

app_server = "cc4f60a8-1f31-ee53-d88b-403eda671809"
vpc_ids = {
  "production" = "5f6dd161-90ed-1b1b-7d7f-046be2e59622"
  "staging" = "6b7f90ae-73b6-e693-25e1-a661428b3cfc"
}

The same for expression works for count modules with a list instead of a map: [for net in module.network : net.vpc_id]. If you need the values as a list from a for_each module, wrap it: values(module.network)[*].vpc_id.

Module sources: pin a Git branch, tag or commit

Local paths are fine inside one repository. When several repositories share a module, keep it in Git and point source at it. The ref query parameter selects what to check out:

module "network" {
  # a release tag: what production should use (if your tags are never moved)
  source = "git::https://github.com/acme/terraform-modules.git//network?ref=v1.4.0"
}

module "network_dev" {
  # a branch: moves every time someone pushes, fine while you develop the module
  source = "git::https://github.com/acme/terraform-modules.git//network?ref=feature/ipv6"
}

module "network_pinned" {
  # a full commit SHA: cannot move at all
  source = "git::https://github.com/acme/terraform-modules.git//network?ref=4f2a9c1e8b7d6a5f4e3d2c1b0a9f8e7d6c5b4a39"
}

Details that trip people up:

The double slash (.git//network) selects a subdirectory of the repository. Without it, Terraform uses the repository root.
Private repositories over SSH use git::ssh://git@github.com/acme/terraform-modules.git//network?ref=v1.4.0, or the shorter git@github.com:acme/terraform-modules.git//network?ref=v1.4.0. The machine running Terraform (including CI) needs a key that can read the repository.
Branch sources do not update on their own. A repeat terraform init keeps the modules already in .terraform/modules as long as their source is unchanged. To pick up new commits on the branch, run terraform get -update (modules only) or terraform init -upgrade (modules, and it also reconsiders provider versions). A fresh checkout, like a CI runner, downloads whatever the branch points at right now. That is exactly why production should not point at a branch: two runs of the same commit of your root configuration can use different module code.
Tags can move. Git lets someone delete a tag and create it again on another commit. If nobody in your organisation re-tags releases, a tag is a good pin; if you cannot be sure, pin the full commit SHA.
Version constraints (version = "~> 1.4") only work with registry sources, not with Git URLs. With Git, the ref is your version pin.

"Provider configuration not present" when you refactor modules

This one surprises people because it appears after deleting code, not adding it. Older modules often declared their own provider block inside the module. Here is one:

# modules/legacy/main.tf
terraform {
  required_providers {
    random = { source = "hashicorp/random" }
  }
}

# a provider block inside a child module: the legacy pattern
provider "random" {}

resource "random_pet" "name" {}

Everything works until you remove the module "legacy" block from the root to get rid of it. Terraform wants to destroy random_pet.name, but the provider configuration it needs to do that lived inside the module you just deleted:

removing a module that had its own provider block

$ terraform plan
Error: Provider configuration not present
To work with module.legacy.random_pet.name (orphan) its original provider
configuration at
module.legacy.provider["registry.terraform.io/hashicorp/random"] is required,
but it has been removed. This occurs when a provider configuration is removed
while objects created by that provider still exist in the state. Re-add the
provider configuration to destroy module.legacy.random_pet.name (orphan),
after which you can remove the provider configuration again.

The fix is to separate the two changes. First move the provider configuration to the root and delete the provider block from the module. Child modules inherit the root's default provider configurations automatically, so the module keeps working. Apply that on its own:

# main.tf (root)
provider "random" {}

module "legacy" {
  source = "./modules/legacy"
}

step 1: provider moved to the root

$ terraform plan
No changes. Your infrastructure matches the configuration.
$ terraform apply -auto-approve
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
# step 2: now delete the module block
$ terraform plan
  # module.legacy.random_pet.name will be destroyed
  # (because random_pet.name is not in configuration)
Plan: 0 to add, 0 to change, 1 to destroy.

Now removing the module is a normal destroy. If you want Terraform to forget the resources instead of destroying them, put a removed block in place of the module (from = module.legacy with destroy = false); our Terraform state post covers removed and moved in detail.

Two rules keep you out of this for good:

No provider blocks in reusable modules. A module declares what it needs in required_providers; the root decides how providers are configured.
When a module needs a non-default provider , for example a second AWS region, pass it explicitly from the root:

provider "aws" {
  alias = "eu"
  region = "eu-west-1"
}

module "backup_bucket" {
  source = "./modules/bucket"
  providers = {
    aws = aws.eu
  }
}

For the bigger picture of sharing providers and variables between many modules, see how to share providers and variables across Terraform modules.

Summary

Question	Answer
How do I read a value from a module?	Add an `output` in the module, read `module.<name>.<output>`
Can I reference a resource inside a module directly?	No, only its outputs
How do I pass a resource into a module?	Pass its attributes, or a typed object, through a `variable`
Do chained modules need `depends_on`?	No, references create the dependency
How do I read outputs from a `for_each` module?	`module.x["key"].output`, or a `for` expression over `module.x`
Branch, tag or commit as a Git source?	Tag or commit for shared code, branch only while developing
"Provider configuration not present"?	Move the provider to the root, apply, then remove the module

Modules stay pleasant to work with when their interface is small and explicit: a few typed variables in, a few described outputs out, no provider blocks inside. For the language features that go into those interfaces, Terraform variables, loops and outputs is the next read, and for laying modules out across environments, see how to organize Terraform modules for multiple environments.

Originally published at devops-daily.com.