DEV Community: DevOps Oasis

Docker Tags Make Production Rollbacks Ambiguous

DevOps Oasis — Thu, 27 Aug 2026 17:31:28 +0000

Our production manifests should name bytes, not somebody’s label.

While the registry UI was showing ledger-api:2026.08.3 with two different upload times, I decided we should deploy Docker images by digest, even when a version tag looks perfectly respectable. A tag is a label somebody can move; production should name the exact bytes it intends to run.

A Docker Tag Is A Promise We Cannot Verify Later

We still create tags. Developers need them, release notes need them, and nobody wants to read a digest aloud during an incident unless morale has already collapsed. 2026.08.3 tells a human roughly what they are looking at. sha256:5bb4… tells a machine precisely what it is looking at.

Those are different jobs, and treating one identifier as both has caused more confusion than it deserves.

A Docker image digest identifies a content-addressed manifest. Docker documents the distinction clearly in its image digests reference: tags can change, while a digest points to a specific image manifest. The OCI image specification is equally blunt about descriptors being identified by digest. We knew this, of course. Knowing a thing and arranging work around it are separate departments.

My opinion is that a production deployment manifest should contain registry.example.net/ledger-api@sha256:..., not registry.example.net/ledger-api:2026.08.3. The tag belongs beside it in Git, in the release ticket, and in the message we send when a deploy starts. It does not belong where the scheduler decides what to pull.

The common reply is that version tags are immutable in a well-run registry. That is usually shorthand for “we asked people not to overwrite them.” A permission policy is better. A registry setting that rejects overwrites is better still. Neither changes the fact that the deployment record itself is vague when it names a tag.

If we need to answer “what was running at 02:13?”, we should not need to reconstruct registry history.

Immutable Tags Are Good Advice With A Missing Clause

The usual advice is to use semantic version tags and make them immutable. We did that for a while. Our GitHub Actions workflow built ledger-api:2026.08.3, pushed it to Harbor, and our deployment repository updated the same tag in the Helm values file.

That worked right up to the point it did not.

In February, a build for the ARM64 variant finished after the AMD64 build. The release job had pushed a single-platform manifest under the version tag first, then replaced it with a multi-platform manifest list once the second build arrived. Nobody was trying to smuggle a patched image into production. The workflow was merely assembled in the order that seemed convenient on a Wednesday morning.

The digest changed anyway.

Our production nodes are currently AMD64, so the pods kept working. A staging runner on ARM64 started pulling the newer manifest list. Both environments reported 2026.02.6; they were not consuming the same top-level image object. The difference mattered when an engineer tried to reproduce a staging-only startup failure using the production digest from the cluster event log.

We fixed the build ordering. Then we noticed the larger flaw: even an immutable tag provides a weak deployment record if the tag was resolved at different times by different tools.

Kubernetes has sensible rules here, but they are not a substitute for stating intent. Its image pull policy documentation explains that Always checks the registry on each launch, while the runtime can reuse locally cached layers and manifests as appropriate. If the image reference includes a tag, the kubelet resolves what that tag means when it pulls. If it includes a digest, there is little to interpret.

“Immutable tags” solve accidental replacement. They do not make a tag an exact deployment reference.

Our Release Controller Stopped Guessing In April

We changed the release path for six customer-facing services in April: ledger-api, settlement-worker, webhooks, invoice-renderer, auth-edge, and report-export. GitHub Actions still pushes a readable tag. After the registry has accepted the final multi-architecture manifest, the workflow asks Harbor for its digest and writes that digest into the deployment pull request.

The pull request title still says ledger-api 2026.08.3. The changed line says something less charming:

image: harbor.ops.internal/ledger-api@sha256:5bb4c0e31c4d9f0a

That one line has made incident handling noticeably less theatrical.

On 7 July, invoice-renderer began returning 500 responses for PDFs containing a non-breaking space in an address field. The release had gone out at 10:06. At 10:19, the on-call engineer found the exact digest in the Argo CD application history, opened the prior deployment commit, and reverted one Git change. The replacement pods were ready by 10:27.

Previously, our rollback notes said things like “back to 1.18.4.” This looks clear until someone asks whether 1.18.4 means the original image, the image rebuilt after a base-image CVE, or the one an engineer pushed locally during a failed release rehearsal. We had all three at one point. No amount of confident punctuation fixes that.

The registry tag now has a narrower purpose. It is a discovery handle: find the release, find the digest, inspect the provenance. We attach the Git commit, build run URL, SBOM location, and source tag as OCI annotations. The OCI annotations guidance gives us standard fields for much of this, though we also carry two internal labels because apparently every organisation eventually invents metadata nobody else wants.

This only held because our deployment repository already owned the image reference. Teams using a CI system that applies directly to a cluster have a harder migration. There is no pull request in which to review the digest, and a shell history is a poor change record.

The Best Objection Is That Digests Hurt Humans

The strongest argument against digest-pinned Docker deployments is that they make ordinary work worse.

That argument has teeth. A digest cannot tell us whether it contains version 3.12.1 or 3.12.2 of an application. It cannot tell us which branch produced it. If a developer sees a 71-character hash in a Kubernetes event, they need another system to map it back to the release. A version tag is readable in a way hashes never will be, and pretending otherwise is how we end up making people keep private spreadsheets.

Digests also complicate emergency changes. If an engineer needs to deploy a temporary image at 01:40, asking them to look up a manifest digest, edit a Git file, wait for reconciliation, and check policy output can feel like process taking a victory lap around a burning bin.

We have done the 01:40 version of this. The Docker registry was healthy, the cluster was healthy, and the thing slowing us down was our own release machinery rejecting an image because the SBOM upload had lagged by four minutes. The service had a memory leak, not a compliance emergency. Still, nobody on the call found the delay spiritually enriching.

There is also a real security concern in the opposite direction. A digest pins an image, including a vulnerable one. A floating tag can pick up a patched base image when a pod restarts. That behaviour is uncontrolled, but it may reduce exposure in a narrow case. Pinning forces us to deliberately rebuild and redeploy, which is operationally correct and sometimes operationally inconvenient.

I still prefer the inconvenience. Silent changes in production are worse than explicit changes arriving later than we would like.

Digest Pinning Fails For Some Kinds Of Docker Work

We do not pin every Docker reference by digest.

Local development is the obvious exception. Our compose file uses postgres:16.4, redis:7.4, and a few internally tagged service images because the point is to get a working stack onto a laptop before lunch. Forcing every developer to update a digest for every base-image refresh would create a small administrative cult and very little safety.

Ephemeral preview environments get tags too. A pull request deployment named pr-1847 is intentionally disposable, and its tag tells us more than a hash does when we are cleaning up fifty previews after a busy week. We retain the build metadata, but we do not pretend those environments have the same change-control needs as production.

Base images are messier still. We pin our production Dockerfiles to digests for cgr.dev/chainguard/node, but that means Renovate opens digest update pull requests several times a week. Priya, who owns most of our runtime maintenance, thinks this is a fair price for seeing every base-image movement. I think the PR volume is irritating, although I have not volunteered to triage it, so my objection has limited moral force.

We have not tried digest-based deployment past 40 nodes or across separate regional registries. Our largest cluster has 28 worker nodes, and Harbor sits on the same internal network as it. If a team relies on registry mirrors with uneven replication, a digest may expose propagation mistakes that a tag previously hid. Hiding the mistake was not a solution, obviously, but an unavailable deployment is still unavailable.

There is one unresolved problem in our setup: old digest retention. We want enough history to roll back an image from six months ago, but storing every release, failed build, and preview manifest has pushed Harbor’s object storage bill up 31% since January. We are arguing over whether retention should follow deployment history or vulnerability status. Those rules produce very different bins of images.

Keep Tags For People And Digests For Schedulers

The practical version of this policy is less dramatic than it sounds.

Build an image once. Tag it with a release version, a Git SHA, and whatever temporary CI name helps the workflow. Publish the final manifest only after every required architecture is present. Read the digest from the registry after that publication, rather than calculating it from a local build result. Then place the digest in the production deployment definition.

Keep the readable information close by. Our release pull requests include the version tag, commit SHA, previous digest, new digest, and a link to the registry artifact. Argo CD shows the digest in the application history. Our #release-watch message includes both references, because people searching Slack will type 2026.08.3, not 64 hexadecimal characters.

We also block tag-only production images through admission policy. We chose Kyverno for this because the policy reads like something a tired engineer can inspect during an incident. The initial rollout was report-only for two weeks. It found three CronJobs and a forgotten namespace called data-scratch, which had survived two reorganisations and a platform migration through what I can only call administrative fog.

The policy has an escape hatch for an incident commander, valid for two hours and logged to our audit channel. We have used it once. The follow-up action was not “remove the guardrail”; it was “make the release workflow stop waiting for an irrelevant test suite.”

I would rather explain a long digest during a rollback than explain why the same Docker tag meant different images on different days. We still need to settle retention before the storage bill becomes its own ticket queue.

Terraform State Locking During Tuesday’s IAM Cleanup

DevOps Oasis — Thu, 27 Aug 2026 17:25:21 +0000

One stale lock, one real permission gap, and a policy we still dislike.

09:12.

That was when stand-up stopped being stand-up and turned into three people staring at a Terraform lock held by a GitLab runner that had supposedly finished forty minutes earlier. Someone had brought a box of almond croissants, which made the delay feel less hostile than it deserved.

09:12 — Stand-Up Opens On A Locked State

The ticket was small enough to sound harmless: remove the remaining broad access from the data-export role in our production analytics account. Finance had moved the nightly export job to a narrower S3 prefix two weeks earlier, and the old policy still allowed reads across the whole reports bucket.

We’d already approved the pull request on Monday. The plan had shown one IAM policy replacement and no resource destruction. By our standards, it was almost suspiciously tidy.

Then the scheduled apply job failed while acquiring the state lock. The Terraform state for this account still lives in an S3 backend with DynamoDB locking. We know newer backend options exist. We also know that “we should migrate state locking later” has been on our board since November, sitting beside “replace the office chair that squeaks during incident calls.”

A failed lock is not automatically a stale lock. That distinction matters more than people think. An engineer running a local apply from a train Wi-Fi connection is annoying. Two applies changing IAM policies against the same state are a much better way to spend Tuesday explaining ourselves to auditors.

We checked the GitLab job page first. It had not failed cleanly. The runner had lost contact during the apply stage, then sat in a “running” state long enough for GitLab to declare it dead. The EC2 runner instance had been replaced overnight by its autoscaling group. The lock remained.

Nobody touched force-unlock during stand-up. We’ve earned that restraint.

09:40 — We Find The State File Everybody Shares

By 09:40, we had the lock ID, the runner hostname, and the commit SHA from the previous night. The useful question was whether Terraform had made any changes before the runner disappeared.

Our analytics account state is larger than we’d like: 186 managed objects, including IAM roles, S3 bucket policies, EventBridge schedules, a few Glue jobs, and the KMS grants that always become someone’s problem eventually. It grew this way because “analytics” was a convenient boundary when there were two data jobs and one AWS account. We now have five account environments and a state file that makes reviewers scroll for longer than is healthy.

We checked CloudTrail for IAM policy changes around the runner’s last heartbeat. Nothing. We checked the S3 backend version history. No new state object version. The runner had acquired the lock, downloaded state, and disappeared before it changed AWS or wrote anything back.

That gave us enough confidence to remove the lock manually. Terraform’s own state-locking documentation is sensible on this point: locks prevent concurrent writes; they do not tell us whether a missing process was harmless. We still need evidence from the provider side.

We keep a short runbook for this. Not a twelve-page document with colour-coded arrows. Just the account, backend location, lock table, who can clear a lock, and the checks required before doing it. The last line says: “A green GitLab page is not evidence.” That line arrived after a green GitLab page turned out to be reporting an old runner status from a job that had already vanished.

10:25 — The Plan Finds A Permission We Had Missed

With the lock cleared, we ran the approved plan again from a fresh runner. The policy change itself was plain: remove a bucket-wide s3:GetObject permission and retain access to reports/finance-export/*.

The second plan differed from Monday’s plan.

Terraform wanted to add s3:ListBucket with a prefix condition. Monday’s version had not. We hadn’t changed the policy file. The difference came from a module release merged by another team earlier that morning. Their update corrected how the module built conditions for prefix-scoped S3 access. Our configuration picked it up because we had allowed a minor module version range.

This is why we pin Terraform providers and have a terraform.lock.hcl file, but we are less relaxed about internal module ranges than we used to be. Provider dependency locks are explained well in HashiCorp’s dependency lock file documentation. Modules need the same degree of suspicion, even when they live in our own GitLab group and have a reassuring name like aws-iam-basics.

The added permission was correct. An S3 client that reads objects often needs to list the prefix first, depending on how it discovers keys. Removing bucket-wide reads while leaving the job unable to list its approved prefix would have made the nightly export fail at 02:00, which is an unpleasant way to prove least privilege.

We paused the apply and asked the data engineering team to confirm the export’s call pattern. They replied with a CloudTrail event from last week: ListBucket, then GetObject against a timestamped CSV under the expected prefix. Good. Concrete evidence beats everybody’s memory of what a Python script “probably does.”

AWS’s policy evaluation logic is worth reading whenever an IAM change feels obvious. Explicit denies, identity policies, resource policies, permission boundaries, session policies: it is entirely possible to be right about the line you removed and wrong about the permission path that remains.

11:50 — We Leave The Provider Upgrade Alone

Before lunch, somebody noticed the runner image carried Terraform 1.9.8 while our newest platform image had 1.10.5. The AWS provider was at 5.72.1 in this repository, though several newer versions were available.

This produced the usual suggestion: while we are in here, should we update everything?

No.

We dislike opportunistic Terraform upgrades during access changes. A provider upgrade can alter a plan in ways that have nothing to do with the ticket in front of us. Sometimes that is exactly the point of upgrading. It deserves its own pull request, its own review, and preferably an afternoon where the person doing it does not also have pager duty.

Terraform plans are useful, but they are not contracts with AWS. They are predictions from a specific Terraform version, provider version, state snapshot, and set of credentials. Change one part of that set and we need to read the prediction again. The plan command reference says plenty about modes and options; the operational bit is simpler: save the plan we reviewed, then apply that plan from an environment we control.

This only held because our applies come from a small pool of disposable GitLab runners, all using the same container image and IAM role. We have not tried this setup across forty separate teams with their own runner images, and we do not pretend the habits would survive unchanged.

Lunch was reheated noodles eaten while someone explained why their new mechanical keyboard had “thock.” We agreed it made a noise.

13:20 — The Apply Produces One Useful Scare

Data engineering confirmed the prefix requirement just after one. We accepted the module change, regenerated the plan, and had the reviewer look at the rendered IAM document rather than trusting the resource count.

The apply completed in 38 seconds. We kept the terminal output in the ticket because the first failure and the successful retry told a clearer story together than a green pipeline badge did.

$ terraform apply tfplan-2026-08-27-1314

Acquiring state lock. This may take a few moments...
aws_iam_role_policy.data_export: Modifying... [id=data-export:read-finance-reports]
aws_iam_role_policy.data_export: Modifications complete after 3s [id=data-export:read-finance-reports]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Outputs:

data_export_role_arn = "arn:aws:iam::482761903155:role/data-export"

The useful scare came five minutes later. The export job’s pre-flight check failed in staging because its assumed-role session policy still allowed the old path but did not include the new list action. Terraform had done precisely what we asked. Our staging job wrapper had quietly supplied an extra session policy, which is the sort of detail nobody remembers until it blocks an otherwise correct change.

We fixed that in the job repository, not in Terraform. Giving the base IAM role broader access to compensate would have been lazy. Session policies are where we want this restriction, because the export job is the only caller that needs it.

By 13:48, the staging run had listed the prefix, read its test object, and exited. Production followed after the normal approval window.

14:35 — Review Turns Into A State Boundary Argument

The reviewer’s main comment was not about IAM. It was about why a data export policy shares state with Glue jobs, KMS grants, and EventBridge rules that have nothing to do with one another.

Fair.

We have been arguing for months about whether to split the analytics state by service ownership or by change frequency. Our preference is change frequency. IAM and bucket policies change in small, review-heavy increments; Glue jobs tend to move with application releases; KMS grants barely change until they suddenly ruin someone’s week. Putting them all in one state means an IAM cleanup takes a lock on work unrelated to IAM.

The opposing view is that splitting state produces more remote-state references, more permissions, and more ways to apply components in the wrong order. Also fair. Terraform dependency graphs are pleasant inside one state file and much less charming once we start wiring outputs across repositories.

We still think the current file is too broad. We do not think “one state per microservice” is an answer. That way lies 70 tiny repositories, each with a README last touched in 2024 and an owner who moved teams.

The unresolved part is KMS. Its key policies and grants cut across enough services that no proposed boundary looks clean. We can move the export role and bucket policy into their own state. We have not agreed where the encryption key belongs.

16:10 — Pager Duty Gets Dragged Into Terraform Again

At 16:10, our on-call engineer asked why they had needed to join a state-lock investigation for a change they did not own. It was a fair complaint, especially since they were already dealing with an alert from a flaky DNS health check.

Our current rule says the person on pager can approve emergency infrastructure changes, but should not be the only person clearing a production state lock. The reason is boring: lock removal has enough blast radius that tired judgment is a poor control. The irritating part is that our access model makes the on-call role one of only four roles allowed to do it.

We need to separate emergency visibility from emergency mutation. Right now, the same permission set grants read access to backend state metadata and the ability to remove a DynamoDB lock. That was convenient when the team was smaller. It now means an on-call engineer gets pulled into a routine recovery step merely because the rest of us designed the permissions around convenience.

We added a ticket to create a narrow break-glass role with audit logging and a short session duration. It will probably sit next to the state-splitting work for a while. Terraform backlog items reproduce when left alone.

17:05 — We Should Automate The Evidence, Not The Unlock

The part of this day worth automating is the evidence gathering after a failed apply: runner status, last state version, relevant CloudTrail events, and whether any resource changed after the lock was created. A bot can collect those facts into the incident ticket in under a minute.

We should not automate force-unlock just because the bot gets impatient.

The coffee was cold by then.

We can make stale-lock recovery less manual, but a human should still decide whether an absent process is truly absent. The lock is there to make us stop before two people write different stories into the same state file, and we have enough of those already.

Scrum Myths We Retired After Thursday Spillover

DevOps Oasis — Sun, 23 Aug 2026 13:33:29 +0000

What our platform team learned when sprint planning met production work.

By 15:17 on Thursday, six of the nine cards we had promised for the sprint were still open, and the one everyone wanted finished was waiting on a DNS change from another team. Our board looked less like a plan and more like the contents of a drawer after someone has searched for a battery.

We’d been running Scrum with the infrastructure group for four months: eight engineers, one engineering manager, a rotating facilitator, and enough Kubernetes work to make estimates feel mildly dishonest. We kept the ceremonies small, used two-week sprints, and made every operational interruption visible instead of pretending PagerDuty events arrived from another department.

That last part mattered. So did admitting that some things people say about Scrum are true only in the same sense that “the lift is working” is true while it is stuck between floors.

“Scrum Makes Delivery Predictable”

Verdict: mostly false, unless the work is already fairly predictable.

Our first three sprints did make delivery look more predictable. We averaged 31 points, then 34, then 32. Somebody put the numbers in a slide. Somebody else drew a trend line. We all briefly enjoyed the illusion that we had discovered arithmetic.

Sprint four contained a certificate rotation, a flaky EKS node group, two urgent access requests from finance, and a production issue where orders-worker kept opening connections after its PostgreSQL failover target had changed. We completed 14 points. The number was not the problem; the work was. Half of it had arrived after planning, and the other half had hidden dependencies behind apparently sensible tickets.

Scrum can force us to state an expectation before work begins. That is useful. It does not turn unknown work into known work, and it certainly does not persuade a cloud provider to schedule maintenance around our sprint boundary.

The Scrum Guide is careful here: a Sprint Goal is an objective, not a contract for a fixed pile of tickets. Teams tend to forget this because tickets are easier to count than objectives are to discuss. “Improve the reliability of cluster upgrades” sounds vague in a planning meeting. “Complete PLAT-184 through PLAT-191” feels reassuring, right up until PLAT-187 reveals a three-year-old Terraform module nobody has touched since 2023.

We changed our planning habit after that Thursday. We now select one Sprint Goal, reserve capacity for known operational work, and mark confidence on items with outside dependencies. A card waiting on Security, Data Engineering, or a vendor is not “in progress” in the same way as a card with a developer actively changing a file. This sounds obvious because it is obvious. We had still failed to do it.

Our predictability got worse before it got better. In May, the percentage of planned work completed fell from 82% to 54% when we stopped quietly pulling unfinished cards into the next sprint and started recording interruption work properly. Finance did not love the first report. Neither did we. It also showed that on-call work was consuming about 11 engineer-days a month, which was difficult to ignore once the number had a row in it.

Four sprints later, our completion rate settled around 73%, but the forecast conversations became less fictional. We can usually say whether a change will land in the next two weeks. We do not promise it when an external team holds the last key.

That is about as predictable as infrastructure work gets, and we are happier with it than the old 90% figure built from invisible spillover.

“Every Sprint Needs A Finished Commitment”

Verdict: false, and this wording causes more trouble than it saves.

We tried treating the sprint backlog as a commitment in the contractual sense. At planning, every card was assigned, estimates were discussed until people began using phrases like “relative complexity” with straight faces, and the team left with a tidy board.

Then Friday came.

A committed backlog produces bad behaviour when reality moves. Engineers split cards late so something can reach Done. They avoid useful investigation because it threatens the count. Work that should be stopped carries on because abandoning it looks worse than spending another day on it. We saw all three during a migration from an old NGINX ingress setup to AWS Load Balancer Controller 2.7.

One ticket was to move the customer-admin service. It had a seven-point estimate and a sensible acceptance list. During testing, we found that its websocket timeout had been set in an annotation inherited from a chart no one remembered choosing. The right response was to pause, reproduce the failure under load, and decide whether to move the service at all.

Instead, we kept pushing. The ticket crossed the board on the last afternoon. It was “done” because the service answered HTTP requests in staging. The websocket issue returned in production at 09:12 on the following Monday. We had manufactured a green card and bought ourselves a support incident.

A Sprint Goal can be a commitment. “Reduce the risk of expiring internal certificates before the September renewal window” gave us room to drop an unhelpful automation task and spend two days fixing the actual renewal path. The goal remained valid even when a ticket did not.

We now call individual backlog items forecasts. Some people dislike the word because they think it softens accountability. We think it describes reality. An engineer still owns the work. The team still explains why it moved or did not move. Nobody gets to call a ticket “almost done” for three sprints and hide behind uncertainty.

The distinction also makes review less ceremonial. We look at what changed for users or operators, what we learned, and what is parked. If a ticket has been open for more than ten working days, we ask whether it is too large, blocked, or simply no longer worth doing. That question has killed several pieces of work that would otherwise have become permanent residents of the board.

We did try a “commitment buffer” of 20%, where every sprint had deliberately unallocated capacity. It did not work. Urgent work filled it by lunchtime on day two, and people started treating the remaining 80% as a harder promise. We removed it after two sprints. No grand lesson emerged. It was just an awkward number sitting on a board while the pager went off.

“The Daily Scrum Is A Status Meeting”

Verdict: false, although many teams have made it one.

Our daily Scrum used to happen at 09:30, last 17 minutes, and contain a sequence of engineers reporting to a manager who already had access to the board. We called it a stand-up despite everyone being on video. It was not offensive; it was merely pointless.

The manager stopped attending after we agreed that the meeting belonged to the people doing the sprint work. This was not a rebellion against management. There are enough of those already, usually involving spreadsheets. It was a decision to stop turning coordination into a recital.

The format changed with one rule: talk only about the next 24 hours of progress toward the Sprint Goal, including anything likely to prevent it. “Yesterday I updated the Helm chart” is status. “The chart deploys, but the admission controller rejects our security context and I need Mina for 20 minutes after this” is coordination.

That took some practice. The first week was full of habit. People gave the old three questions, then added a blocker as an afterthought. By week three, the useful discussions had become shorter and more direct. One engineer would say a task had expanded; another would point out a similar change in payments-api; someone would decide to pair after the meeting rather than spend two days rediscovering the same failure.

Our daily Scrum now runs for nine minutes on most days. On Mondays it can run longer because somebody has usually found a surprise over the weekend. If a production incident is active, we cancel it and use the incident channel instead. There is no prize for maintaining a ceremony during a fire.

We also stopped forcing every person to speak. Silence is allowed if someone has no change to coordinate. This makes a few Scrum purists twitch, but the guide’s purpose for the event is inspection and adaptation, not attendance theatre.

There is one unresolved argument here. We rotate facilitation every sprint, and two engineers still think the daily meeting becomes noticeably worse when the rotation reaches people who dislike meetings. They may be right. We have not yet decided whether a team-owned practice needs a consistent owner to stay useful, or whether that is just us avoiding a mildly uncomfortable skill gap.

“Velocity Tells Us When To Hire”

Verdict: false, and we dislike velocity as a management number.

Velocity tells us how many points a particular team finished under a particular estimating habit. It does not tell us how much work exists, whether the work is valuable, whether another engineer would remove a bottleneck, or whether the points themselves are anything more than educated hand gestures.

We learned this when our eight-person platform group’s velocity rose from 29 to 41 across five sprints. The headcount had not changed. The work had. We had spent a month closing small access-control tasks and documentation gaps after a large migration. The average card got smaller. Our estimate scale drifted because people had become familiar with the code. No new capacity had appeared, unless confidence in kubectl counts as capacity.

A senior manager asked whether the data showed we could defer an open platform role. We said no, then had to explain why a rising line did not mean the team was suddenly 41% more capable. That conversation took longer than it should have, partly because we had supplied the line in the first place.

For staffing, we now bring actual constraints: time spent on on-call, the number of services per primary responder, lead time for changes requiring platform review, and the queue of work we have agreed is worth doing. The DORA research is more useful than sprint velocity when discussing delivery performance, though it will not tell us whether we need someone who understands IAM policy evaluation at 02:00.

Our cost story was equally unhelpful at first. We added a contractor to cover the observability backlog, and monthly cloud spend rose by $2,840 before it fell. The contractor found duplicated OpenSearch ingestion and an abandoned Datadog log pipeline, but the cleanup took six weeks and required running old and new paths together. The early finance report showed only a person costing money and a larger AWS bill. By July, the recurring spend was down by roughly $1,900 a month. Velocity did not explain any of that.

We still estimate. Planning with no rough sizing tends to turn every ticket into a philosophical dispute. But we keep points inside the team, refuse comparisons between teams, and do not put velocity in hiring cases. A hiring decision based on story points is one meeting away from measuring engineers in bananas.

“Scrum Needs Someone To Protect The Team”

Verdict: mostly true. We wish it were less true.

We have seen teams say they are self-managing, then accept every Slack request, executive question, customer escalation, and “quick” production change because nobody has the standing to say no. The result is a sprint plan that exists mainly to be apologised for on the final Friday.

Someone has to protect the time reserved for the work the team agreed mattered. In our case, that person is usually the engineering manager, but it is not exclusively their job. The rotating facilitator flags new requests during daily Scrum. The on-call engineer can redirect a non-urgent ask into the intake queue. Staff engineers push back on technically dangerous shortcuts. Protection works better when it is shared, because a manager in a meeting cannot stop a drive-by request in a Slack thread.

Still, a named person needs enough authority to resolve the hard cases. Last month, Sales wanted a one-off IP allowlist for a prospect’s proof of concept. The request arrived eight business days before a security hardening deadline. The technical change would have taken perhaps two hours. The review, exception handling, expiry process, and later cleanup would not. Our manager said no until the Sprint Goal was met. The prospect did not vanish. The certificate work finished on time.

This only held because our director backed the decision. On teams where every request can be escalated around the manager, “protect the team” becomes a polite phrase for losing an argument more slowly.

We do not need a full-time Scrum Master for eight engineers. We do need somebody prepared to tell a vice president that their urgent request is a backlog item until they can explain what it displaces.

Kubernetes NetworkPolicy Rules We Can Actually Prove

DevOps Oasis — Sun, 23 Aug 2026 13:29:25 +0000

Build a locked-down namespace, then test every allowed connection ourselves.

The orders-api namespace was accepting traffic from a forgotten batch job at 08:46, which was annoying because we had already written “internal only” in three tickets. Kubernetes had done exactly what we told it to do: nothing. Pods can generally talk to other pods unless the network plugin says otherwise.

We fixed the small version of that problem first. By the end of this walkthrough, we’ll have an orders-api namespace where:

the API accepts port 8080 traffic only from its frontend;
the API can reach PostgreSQL on 5432;
the API can resolve DNS;
random pods cannot connect to the API or send arbitrary outbound traffic.

This uses standard NetworkPolicy objects. We tested the same manifests on Kubernetes 1.30 with Cilium 1.16. Calico works too, but do not assume every cluster’s CNI enforces policies. Kubernetes will happily accept a NetworkPolicy object that changes precisely nothing. A very Kubernetes outcome.

Check Whether The Cluster Enforces NetworkPolicy

First, point kubectl at a cluster we can safely alter.

# Confirm the current context before creating anything.
kubectl config current-context

# Confirm the Kubernetes server version.
kubectl version

# See which CNI components are running.
kubectl get pods -n kube-system -o wide

On our clusters, Cilium pods show up as cilium-xxxxx in kube-system. If we’re using Calico, we expect calico-node-xxxxx. The exact pod names are less interesting than whether the plugin supports policy enforcement.

Check the plugin documentation rather than trusting a chart value from six months ago. The relevant references are Cilium’s Kubernetes NetworkPolicy docs and Calico’s policy documentation.

For a disposable local cluster, we use kind with Cilium installed separately. This is optional if there’s already a policy-capable cluster available.

# Create a local Kubernetes cluster for the exercise.
kind create cluster --name networkpolicy-lab

# Install the Cilium CLI if it is not already installed.
# macOS example:
brew install cilium

# Install Cilium and wait until its agents are ready.
cilium install
cilium status --wait

# Point kubectl at the new kind cluster.
kubectl config use-context kind-networkpolicy-lab

Don’t move past this step on faith. A policy test on a cluster with no enforcing CNI gives us a false sense of safety, which is worse than no test because somebody will cite it during an incident review.

Create The Three Workloads We Need

We need an API, a frontend client, and a PostgreSQL-shaped target. The database is only postgres:16-alpine; we are testing network paths, not writing an orders system before lunch.

Save this as 00-workloads.yaml.

# 00-workloads.yaml
---
apiVersion: v1
kind: Namespace
metadata:
  name: orders-api
  labels:
    team: commerce
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: orders-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
        role: backend
    spec:
      containers:
        - name: api
          image: hashicorp/http-echo:1.0
          args:
            - "-listen=:8080"
            - "-text=orders-api is alive"
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: api
  namespace: orders-api
spec:
  selector:
    app: api
  ports:
    - name: http
      port: 8080
      targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  namespace: orders-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
        role: frontend
    spec:
      containers:
        - name: frontend
          image: curlimages/curl:8.10.1
          command: ["sleep", "3600"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  namespace: orders-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
        role: database
    spec:
      containers:
        - name: postgres
          image: postgres:16-alpine
          env:
            - name: POSTGRES_PASSWORD
              value: not-a-real-password
          ports:
            - containerPort: 5432
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: orders-api
spec:
  selector:
    app: postgres
  ports:
    - name: postgresql
      port: 5432
      targetPort: 5432

Apply it and wait for the deployments.

# Create the namespace, deployments, and Services.
kubectl apply -f 00-workloads.yaml

# Wait for all three deployments to become available.
kubectl rollout status deployment/api -n orders-api
kubectl rollout status deployment/frontend -n orders-api
kubectl rollout status deployment/postgres -n orders-api

# Keep this output around; labels drive the policy selectors later.
kubectl get pods -n orders-api --show-labels

We deliberately use workload labels such as app: frontend rather than pod names. Pod names change. Labels are the contract here, so they deserve the same review attention as an IAM role binding.

Prove The Namespace Starts Wide Open

Before adding any policy, both the frontend and an unapproved pod should reach the API. Create the unapproved pod in a separate namespace so we prove that cross-namespace traffic is also open.

# Create a namespace representing an unrelated workload.
kubectl create namespace reporting

# Start a temporary curl pod and leave it alive for testing.
kubectl run reporter \
  --namespace reporting \
  --image=curlimages/curl:8.10.1 \
  --command -- sleep 3600

# Wait until the test pod has an IP address.
kubectl wait \
  --namespace reporting \
  --for=condition=Ready pod/reporter \
  --timeout=90s

# Confirm the approved frontend can call the API.
kubectl exec -n orders-api deploy/frontend -- \
  curl --connect-timeout 3 -sS http://api.orders-api.svc.cluster.local:8080

# Confirm the unrelated reporter can also call it before policy exists.
kubectl exec -n reporting pod/reporter -- \
  curl --connect-timeout 3 -sS http://api.orders-api.svc.cluster.local:8080

Both commands should print orders-api is alive. If the second one already fails, stop and find out why. Perhaps there is an existing global policy, a service mesh rule, or someone has been more diligent than we expected. Copying a baseline test from a blog into a cluster with unknown controls is how we manufacture confusing evidence.

Deny All Inbound Traffic To The API

NetworkPolicy becomes active for a pod in a direction when a policy selects that pod and includes that direction. This policy selects API pods and says all ingress is denied unless another policy permits it.

Save this as 01-api-default-deny.yaml.

# 01-api-default-deny.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-deny-all-ingress
  namespace: orders-api
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Ingress

Apply it, then repeat both checks.

# Turn on ingress isolation for API pods.
kubectl apply -f 01-api-default-deny.yaml

# The frontend should now fail with a connection timeout.
kubectl exec -n orders-api deploy/frontend -- \
  curl --connect-timeout 3 -sS http://api.orders-api.svc.cluster.local:8080

# The unrelated namespace should fail too.
kubectl exec -n reporting pod/reporter -- \
  curl --connect-timeout 3 -sS http://api.orders-api.svc.cluster.local:8080

The curl exit code should be non-zero, usually 28 for a timeout. A rejected TCP connection can also happen depending on the CNI. What we should not see is the response body.

Keep the timeout short in these checks. A default curl timeout during a failed policy test feels like waiting for a kettle that has been unplugged.

This is the shortest section because the object itself is mercifully small. The next policy puts back exactly one route.

Allow Only Frontend Requests On Port 8080

The API needs inbound HTTP from frontend pods in the same namespace. We specify both the namespace selector and the pod selector. Omitting namespaceSelector is easy here because the policy lives in orders-api, but we prefer being explicit in security controls. It makes a later copy into another namespace less surprising.

# 02-api-allow-frontend.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-allow-frontend-http
  namespace: orders-api
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: orders-api
          podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080

Apply it and run the same calls again.

# Permit the frontend-to-API path only.
kubectl apply -f 02-api-allow-frontend.yaml

# This should return the API response now.
kubectl exec -n orders-api deploy/frontend -- \
  curl --connect-timeout 3 -sS http://api.orders-api.svc.cluster.local:8080

# This must remain blocked.
kubectl exec -n reporting pod/reporter -- \
  curl --connect-timeout 3 -sS http://api.orders-api.svc.cluster.local:8080

# Inspect the effective intent in Kubernetes, not just the local file.
kubectl describe networkpolicy -n orders-api api-allow-frontend-http

Policies are additive. The deny policy does not override the allow policy; it establishes isolation, and this policy adds a permitted ingress path. We still see people expect “deny all” to win like a firewall rule at the bottom of a list. Kubernetes NetworkPolicy does not work that way, and frankly I prefer the additive model once everyone has been burned by it once.

Restrict API Egress To PostgreSQL And DNS

Now we isolate egress from the API. This catches the overlooked half of many namespace policies: an exploited API that can call every service and every public endpoint is still having a fairly productive day.

First, find the DNS service IP and labels in this cluster. CoreDNS commonly has k8s-app: kube-dns, but check rather than inventing labels at runtime.

# Find the DNS service and its ClusterIP.
kubectl get service -n kube-system kube-dns

# Inspect CoreDNS pod labels for the policy selector.
kubectl get pods -n kube-system -l k8s-app=kube-dns --show-labels

Create the egress policy. The PostgreSQL rule permits TCP/5432 to pods labeled app: postgres in orders-api. The DNS rule permits UDP and TCP port 53 to CoreDNS pods in kube-system.

# 03-api-egress.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-allow-postgres-and-dns
  namespace: orders-api
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: orders-api
          podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

# Enable egress isolation and allow only database plus DNS traffic.
kubectl apply -f 03-api-egress.yaml

# DNS resolution should still work from the API pod.
kubectl exec -n orders-api deploy/api -- \
  wget -qO- http://postgres.orders-api.svc.cluster.local:5432 || true

# TCP connectivity to PostgreSQL should succeed, even though HTTP is nonsense there.
kubectl exec -n orders-api deploy/api -- \
  sh -c 'nc -zvw 3 postgres.orders-api.svc.cluster.local 5432'

# Arbitrary outbound HTTPS should now time out.
kubectl exec -n orders-api deploy/api -- \
  wget -T 3 -qO- https://example.com

The first command may emit PostgreSQL protocol noise because wget is speaking HTTP to a database. That is expected and mildly ugly. The nc command is the useful test; it should report a successful connection.

We have not tried this exact label-based DNS rule past 40 nodes with NodeLocal DNSCache enabled. That setup changes the destination path, and we still have an open argument about whether to standardise it this quarter. If NodeLocal DNSCache is in use, test it from a real pod before declaring DNS allowed.

Add A Default Deny For New Pods

The API is isolated, but a new pod added to orders-api would still have open ingress and egress unless its own policy selects it. We add namespace-wide default denial so new workloads begin closed.

This is our preference for service namespaces. It forces each deployment PR to state its network needs. It also creates more policy files, which is a fine trade when the alternative is guessing why a maintenance job can reach payroll.

# 04-namespace-default-deny.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: namespace-default-deny
  namespace: orders-api
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

# Apply isolation to every current and future pod in orders-api.
kubectl apply -f 04-namespace-default-deny.yaml

# The frontend now has no egress policy, so its API call should fail.
kubectl exec -n orders-api deploy/frontend -- \
  curl --connect-timeout 3 -sS http://api.orders-api.svc.cluster.local:8080

# List every policy now affecting the namespace.
kubectl get networkpolicy -n orders-api

That failed frontend call is correct. We broke it deliberately, in daylight, while we have the terminal and the attention span to repair it.

Restore The Frontend’s Required Egress

The frontend needs DNS and TCP/8080 to the API. Add those two paths, then rerun the test. We use one policy per workload rather than one huge namespace policy because code review stays readable after the fourth service arrives.

# 05-frontend-egress.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: frontend-allow-api-and-dns
  namespace: orders-api
spec:
  podSelector:
    matchLabels:
      app: frontend
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: orders-api
          podSelector:
            matchLabels:
              app: api
      ports:
        - protocol: TCP
          port: 8080
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

# Restore only the frontend's required outbound paths.
kubectl apply -f 05-frontend-egress.yaml

# Approved frontend traffic works again.
kubectl exec -n orders-api deploy/frontend -- \
  curl --connect-timeout 3 -sS http://api.orders-api.svc.cluster.local:8080

# Cross-namespace traffic remains blocked.
kubectl exec -n reporting pod/reporter -- \
  curl --connect-timeout 3 -sS http://api.orders-api.svc.cluster.local:8080

# Show all objects we would commit with the application manifests.
kubectl get networkpolicy -n orders-api

At this point, we would commit the five YAML files beside the Helm chart or Kustomize overlay that owns orders-api. Keeping policies in a separate security repository sounds tidy until an application team changes its port and the policy change waits in another queue. We tried that arrangement in 2024. It produced a lot of polite Slack messages and one very long release.

When This Goes Sideways

If every connection still works after applying api-deny-all-ingress, the CNI probably is not enforcing NetworkPolicy. Check its DaemonSet, its logs, and any managed-cluster setting that enables policy support. kubectl get networkpolicy only proves the API server stored YAML.

If DNS fails after egress isolation, inspect the actual CoreDNS labels and check whether NodeLocal DNSCache is enabled:

# Check CoreDNS labels and look for NodeLocal DNSCache.
kubectl get pods -n kube-system --show-labels | grep -E 'coredns|node-local-dns'

# Inspect DNS configuration from the affected pod.
kubectl exec -n orders-api deploy/api -- cat /etc/resolv.conf

# Review policy events and the API pod's recent state.
kubectl describe pod -n orders-api -l app=api
kubectl get events -n orders-api --sort-by=.lastTimestamp

If a policy seems correct but traffic still fails, test by pod IP as well as Service DNS. Service handling varies by CNI mode, kube-proxy replacement, and whether traffic is observed before or after translation. We do not guess which one applies; we record the cluster version and CNI mode in the incident ticket.

# Get the PostgreSQL pod IP for a direct connectivity comparison.
POSTGRES_IP=$(kubectl get pod -n orders-api -l app=postgres \
  -o jsonpath='{.items[0].status.podIP}')

# Test direct TCP traffic from the API pod.
kubectl exec -n orders-api deploy/api -- \
  sh -c "nc -zvw 3 ${POSTGRES_IP} 5432"

Finally, remove the lab when it has served its purpose. Leave it around and somebody will point a real test at postgres because the service name looked convenient.

# Remove the exercise namespaces and all namespaced resources.
kubectl delete namespace orders-api reporting

# Delete the local kind cluster if we created one.
kind delete cluster --name networkpolicy-lab

DevOps Questions After We Broke The Release Handshake

DevOps Oasis — Sat, 22 Aug 2026 06:07:26 +0000

Answers from the incident where every dashboard looked politely wrong.

The release-api deployment had already been marked complete when the invoice page began returning 503s. The new container was serving traffic, the PostgreSQL migration had committed, and the feature flag was on. A NetworkPolicy added in another repository prevented the new pod from reaching tax-rate-cache.

The application team saw errors, the database team saw a clean migration, and Platform saw green nodes. By the time we put all three facts in one incident channel, 63 deployment messages had buried the one that mattered.

“Is This Actually A DevOps Failure Or Just One Bad Deploy?”

It was a DevOps failure because four teams completed valid local work and nobody owned the release handoff between them.

Calling it “just a bad deploy” would have been convenient. We could have fixed the policy, replayed the release, written a short incident note, and carried on pretending that a green Argo CD application means a service is ready for users. The pod was healthy according to Kubernetes. It was also unable to call a dependency required to render an invoice. Both things can be true, which is why a deployment status alone is a fairly poor witness.

Our old release process had hidden contracts in too many places:

The service repository declared its image and Helm values.
The infrastructure repository held network rules.
Database migrations ran from a separate GitHub Actions workflow.
Feature flags lived in LaunchDarkly, owned by whoever had last touched the feature.
The runbook lived in Confluence, where it had last been edited in February.

We’ve started putting the release dependencies in the service repository, close to the code that needs them. It is not a clever system. It is a file that a human can read during an incident and a pipeline can check before promotion.

release:
  service: release-api
  requires:
    - dependency: tax-rate-cache
      namespace: finance
      port: 8080
      network_policy: allow-release-api-to-tax-rate-cache
    - migration: 2026_08_17_add_tax_region
      minimum_version: "4.12.0"
  feature_flags:
    - invoice_tax_v2
  smoke_test:
    command: "./scripts/check-invoice-preview.sh"
    timeout_seconds: 90

The pipeline does not yet validate every field. It checks that the named NetworkPolicy exists in the target cluster and runs the smoke test after the migration. That alone would have caught Monday’s problem before 11% of invoice-page requests failed.

We dislike using “DevOps” as a department name. In this case, it means the work of making handoffs visible before production discovers them for us.

“Do Developers Really Need Production Access?”

Yes, but they need narrow production access tied to the service they own, not a shared kubectl credential passed around like an office spare key.

For too long, our answer was to give engineers log access and tell them to ask Platform for anything else. That worked while we had six services and one cluster. We now have 34 services, three production clusters, and an on-call rota where the person holding the pager may know the application code better than the person who built the cluster.

The useful access is usually boring:

Read logs, events, and deployment state in the team namespace.
View configured secrets by name, never by value.
Restart a failed deployment through a recorded workflow.
Fetch an approved diagnostic profile during an incident.
Read relevant cloud metrics and traces.

The access we do not hand out includes cluster-wide secret reads, direct production database writes, and unrestricted exec into pods. There are rare reasons for each. Those reasons deserve an incident ticket and a second human looking at the command.

Our Kubernetes groups now map to service ownership. The catalog team cannot casually inspect identity, and Platform cannot claim every namespace is somebody else’s concern.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: catalog-oncall
  namespace: catalog-prod
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log", "events"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets"]
    verbs: ["get", "list", "watch", "patch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: []

The empty pods/exec permission is deliberate. We had one incident in May where an engineer fixed a live configuration file inside a container, then forgot it existed. The next rollout erased it, as containers are designed to do. We spent forty minutes rediscovering the same failure with more coffee involved.

Kubernetes has plenty of sharp edges here, so we keep the policies close to the upstream RBAC model rather than inventing an internal permissions language.

“Why Can’t Platform Just Own Every Deployment?”

Platform can own the deployment path; it cannot own the operational knowledge of every service.

We tested the “Platform approves production” model last autumn. It looked safe on a process diagram. In practice, it put two platform engineers between every product team and a release, including changes that only altered invoice wording or a CSS bundle. The queue grew, people started sending urgent messages, and approval became a ritual performed after somebody had already decided to deploy.

That is gatekeeping with better YAML.

The service team now owns whether its code should go live. Platform owns the tooling that makes the decision visible, repeatable, and reversible: the GitHub Actions templates, the Argo CD projects, the cluster policies, and the audit trail. Security owns the controls that must be true before the workflow can issue production credentials. Those boundaries are written down because informal agreement lasts exactly until the person who remembers it is on holiday.

We use CODEOWNERS for files that change shared deployment behavior. A service team can change its replica count or an application environment variable. A change to a shared ingress class, an organisation-level action, or a production NetworkPolicy requires Platform review.

# .github/CODEOWNERS
/.github/workflows/reusable-deploy.yml  @platform-engineering
/platform/network-policies/            @platform-engineering @security-engineering
/services/catalog/                     @catalog-team
/services/release-api/                 @billing-team

GitHub documents the awkward parts of CODEOWNERS matching and branch protection, including the fact that a file can have one visible owner line while several teams assume they own it. We found two of those in the first review.

The rule is not “Platform stays out of releases.” We still join risky database changes, regional failovers, and anything touching shared ingress. We simply refuse to become the human merge queue for routine service work.

“What Does ‘You Build It, You Run It’ Mean On Pager?”

It means the team that changes a service takes first call for its user-facing failures, with Platform covering failures in the shared runtime.

We had to make that sentence more precise than it sounds. “Run it” had become a polite way to tell application engineers that they owned every alert, including node pressure, expired cluster certificates, and a broken Fluent Bit daemonset. Nobody learns much from a pager at 03:00 that says “the whole platform is vaguely unhappy.”

Our current split is based on the failing layer. If catalog-api returns 5xx responses because of a bad release, Catalog gets paged. If every namespace loses DNS resolution, Platform gets paged. If the application cannot reach Redis because its own connection pool is exhausted, Catalog gets paged; if the Redis service itself is unavailable, the data platform rotation gets paged and Catalog gets an incident notification.

That sounds obvious after writing it down. It was not obvious in PagerDuty, where 19 services had a generic escalation policy called production-critical.

We changed the alert annotation format so it carries an owner, a runbook URL, and a statement of what user action is failing. The last field has reduced some thoroughly unhelpful alerts.

annotations:
  owner_team: catalog
  runbook: https://ops.example.internal/runbooks/catalog-checkout-errors
  user_impact: "Customers cannot add an item to an existing order"
  dashboard: https://grafana.example.internal/d/catalog-api

Mina, who has been on our platform rota longer than the current Grafana folder structure, objected to putting ownership in annotations. Her point was reasonable: labels rot. She was right, so the alert rule now validates the owner against our service catalog during CI. We still have stale runbooks, but now they fail a check instead of waiting quietly for a bad night.

For alert design, we keep returning to the Google SRE Workbook guidance on paging: page for urgent user impact, not every noisy internal symptom. It has saved us from several clever alerts that nobody could act on.

“Are DORA Metrics Going To Become Another Scorecard?”

They will if we publish team rankings, so we do not.

We track deployment frequency, lead time, change failure rate, and time to restore because they help us locate friction in the delivery system. The DORA research is useful as a vocabulary. It does not grant permission to declare that the team with fewer deployments is failing.

Our billing team deploys release-api three or four times a week. The reporting team deploys its batch export service about twice a month, often after waiting for a partner’s sample file. Comparing those numbers is managerial astrology.

Instead, we review metric changes against specific events. In July, median lead time for Catalog rose from 7 hours to 31 hours. The cause was not developer speed. A required image scan had begun waiting behind the same two self-hosted runners that handled integration tests. We added two runners, separated scan jobs from test jobs, and lead time dropped the following week.

Change failure rate exposed a less comfortable issue in Billing. Their rate was 18% over six weeks, mostly because a database migration and application release were approved separately. The failed release-api deployment belongs in that number. We are not using it to tell the team to make fewer changes; we are using it to fund the work of joining the changes safely.

We show rolling twelve-week trends and attach incident links to failed changes. Nobody gets a traffic-light score. If a metric appears in a performance review, people will optimise the number and hide the problem. We have enough experience with that particular species of spreadsheet.

“Should We Put Argo Rollouts In Front Of Every Service?”

We don’t know yet, honestly.

We have Argo Rollouts running for three HTTP services: catalog-api, release-api, and notifications-webhook. The canary step sends 10% of traffic to the new ReplicaSet, waits five minutes, then checks request error rate and p95 latency. If either crosses the configured threshold, it aborts.

strategy:
  canary:
    steps:
      - setWeight: 10
      - pause:
          duration: 5m
      - analysis:
          templates:
            - templateName: checkout-http-health
      - setWeight: 50
      - pause:
          duration: 10m
      - setWeight: 100

The release-api incident would probably have failed its canary analysis. “Probably” matters. Our staging traffic is too thin to produce a meaningful request-error signal, and production canaries only help when the issue appears in the first slice of traffic. A missing network permission would have done so. A quarterly invoice batch failure would not.

We have not tried it past 40 nodes, and our production clusters have 27, 31, and 38 worker nodes. The controller overhead looks modest in our tests. That is not evidence for what happens after the acquisition team’s services arrive, assuming that deal survives the budget meeting.

We also do not want rollout machinery around every workload. A nightly reconciliation Job does not need traffic shifting. Neither does a CronJob that fetches exchange rates at 04:15. For these, a failed run needs a clear alert and an easy rollback of the image tag. Adding canary objects would make the repository busier without reducing the relevant risk.

Argo’s analysis templates are useful because they force us to state what “healthy” means. The hard part is choosing a metric that reflects users rather than deployment optimism.

“How Will We Know The New Release Rules Are Working?”

We’ll know when a service team can explain a failed production release from one pull request, one deployment record, and one incident timeline.

That is our test for the next six weeks. Each production deployment must link its commit, container digest, migration version, feature flags, smoke-test result, and rollback action. We are adding those fields to the release event emitted into OpenTelemetry, then building the incident view from that event instead of asking people to paste screenshots into Slack.

The first version will be ugly. It only held together this week because the billing team already had consistent Git tags and the migration workflow happened to emit a version number. Catalog does not yet do either. We are fixing the release metadata there before we announce a company-wide standard and discover that standards are easiest when somebody else has done the plumbing.

We do not need a 64th deployment message.

The remaining argument is whether a NetworkPolicy change should block a service release until its owning team approves it, or whether the platform team should approve it as shared infrastructure. We have put that decision on Thursday’s architecture agenda, where it will receive the traditional amount of adult supervision.

Kanban After 12 Months Of Interrupt-Driven Work

DevOps Oasis — Sat, 22 Aug 2026 05:41:07 +0000

One year after replacing sprints, our platform team kept the limits and lost the theatre.

On 19 August 2025, we deleted the remaining two-week sprint from our platform board while seven tickets were still marked “in progress.” Two were waiting on product teams, one was a certificate renewal nobody had planned, and one was a fairly ordinary Terraform change that had somehow acquired four assignees.

We had spent six months pretending this was a planning problem. It was an interruption problem.

Our nine-person platform team supports 36 production services, a shared Kubernetes cluster, identity, CI runners, and the collection of internal tools that only become visible when they fail at 03:00. Sprints made the work look orderly for roughly a day and a half. Then an incident, an access request, or a release deadline arrived and the board became historical fiction.

We moved to kanban with a hard WIP limit, an expedite lane, and a promise to stop calling unfinished work “carryover” as if it had merely missed a train. One year later, we’d grade the decision a B. The board changed how we start work and how we argue. It also created chores we had not put on anybody’s capacity plan.

We Expected The Board To Expose Hidden Work

The expected gain was embarrassingly simple: show the work we were already doing.

Before kanban, an engineer could have a sprint ticket, two Slack requests, a deployment rollback, and a “quick look” at a customer’s IAM policy. Only the sprint ticket appeared in the report sent to leadership on Friday. We were measuring planned work against a week that contained a great deal of unplanned work, then acting surprised when the ratio looked bad.

We expected WIP limits to make people finish things before picking up more things. Our original rule was eight active standard tickets across the team, with one separate expedite slot. Eight felt generous. It was based on the number of engineers minus the people we assumed would be in meetings, which is not a calculation we’d recommend preserving in a museum.

We also expected the classes of service to stop being decorative labels. “Expedite” meant production impact or a security deadline with fewer than five days left. “Fixed date” meant a date imposed by something outside our team: a contract renewal, audit evidence, a vendor certificate expiry. Everything else entered the standard queue.

The Kanban Guide is admirably plain about starting with the current way of working. We took that seriously, perhaps because we did not have enough spare energy to redesign the department and install a new board in the same quarter.

Our old sprint plan behaved like a railway timetable after a signal failure: technically still present, increasingly irrelevant.

The first version had three columns: Ready, Doing, Done. We added Blocked on the second day because “Doing” was hiding work that no one could move. That should have told us something about the first version’s intellectual depth.

Tuesday Replenishment Changed More Than Stand-Up

What actually happened was quieter than we expected. Nobody stood up in a meeting and declared a new era of flow. The useful shift came on the first Tuesday when the Ready column held 19 tickets and Doing had reached its limit.

We had 11 active tickets. More precisely, we had 11 tickets marked active; three were waiting for a reply, two had not been touched since the previous week, and one was a task to “investigate service mesh options,” which is a sentence that should trigger an automatic timeout.

The team started asking a different question: what can we finish before taking this? That sounds obvious when written down. In practice, it changed the order of conversations. An engineer working on an OpenTofu module asked the service owner for the missing variable list before beginning a new request. Another closed a stale upgrade card after confirming the vendor had withdrawn the affected release. We stopped treating context switching as evidence of responsiveness.

Our daily check-in dropped from 25 minutes to around 11. We walk the board right to left, starting with blocked items and then work closest to done. We do not go person by person. The person-by-person version had become a polite recital of activity, and everyone knew it.

Throughput settled around 14 completed standard tickets per fortnight, though the number is lumpy enough that we do not use it for individual targets. In October, we completed 23 because several old requests were finally closed. In January, we completed six while rebuilding the GitHub Actions runner group after an image change broke our ARM jobs.

Little’s Law gave us a useful way to discuss that variation: more work in progress generally means longer time to finish, assuming the system is reasonably stable. The formula is short; the hard part is refusing to add “just one more” item when the board is full. The explanation of Little’s Law is more useful than most delivery dashboards we’ve purchased.

This only held because our engineering managers agreed not to insert work directly into Doing. They can request an expedite item. They cannot assign it a person and declare the WIP limit an inconvenience.

The Blocked Column Made Dependencies Public

The surprise nobody predicted was that blocked work would become mildly political.

A ticket marked Blocked tells a different story from a ticket left in Doing. It tells us the platform team is waiting on a security approval, a product decision, or a vendor reply. That visibility is useful, but it also names the place where work has stopped. Some teams welcomed it. One product group asked whether we could use a softer status because “blocked sounds accusatory.”

We did not.

We did change the policy. Every blocked ticket now needs a short reason, the name of the waiting party, and a next follow-up date. “Waiting on product” is not enough. “Waiting on Checkout to choose whether webhook retries should be at-least-once or deduplicated; follow up Thursday” is enough.

By November, the board showed an uncomfortable pattern. Access work was not slow because creating roles in Okta took long. It was slow because requesters submitted a ticket before deciding which people needed access and why. We had built a form that asked for an application name and environment, then left the actual approval chain to comments.

That led to a small but worthwhile change: access requests now require an owner and an expiry date before entering Ready. The number of access tickets did not fall. Their median elapsed time fell from 9.4 business days to 3.1, mostly because we stopped accepting half-formed requests.

The blocked data also gave us evidence during the February incident review. We could show that six “urgent” observability requests had waited on service owners to define an SLO. The Google SRE workbook’s alerting guidance says alerts should connect to a user-facing objective. We had been asking teams for exactly that, then allowing the cards to enter the queue without it.

There is still a social cost. People dislike seeing their team named in a blocked reason, even when the wording is factual. We have not found a better answer than being precise and not turning the board into a blame register.

The Board Needed Someone To Weed It

Nobody budgeted for board maintenance.

Not administration in the abstract. Actual maintenance: closing dead cards, correcting issue types, checking automations, chasing owners, and making sure a ticket moved to Done reflects a deployed change rather than a pull request waiting for approval.

For the first three months, GitHub Projects automation handled enough of this that we thought we had escaped the usual board janitor role. A pull request merge moved the linked issue to Done. New issues with the platform label entered Triage. An incident label put work into Expedite.

Then reality arrived. A merged pull request could deploy to staging and sit there for nine days. Incident tickets kept the label after the incident was closed. A request from Finance inherited platform because someone copied an old issue template. The board was accurate in the same way a kitchen drawer full of old cables is organised: there was a system, once.

Here is how the setup drifted over the year:

# 19 August 2025
workflow:
  columns: [Ready, Doing, Done]
  doing_wip_limit: 8
  expedite_wip_limit: 1
  done_when: "pull request merged"
automation:
  label_platform: "add to Ready"
  label_incident: "move to Doing"

# 19 August 2026
workflow:
  columns: [Triage, Ready, Doing, Blocked, Verify, Done]
  doing_wip_limit: 6
  expedite_wip_limit: 1
  blocked_requires: [reason, waiting_on, follow_up_date]
  done_when: "production verification recorded"
policies:
  triage_owner_rotation: weekly
  stale_ready_after_days: 30
  expedite_requires: "active customer impact or security deadline under 5 days"
automation:
  label_platform: "add to Triage"
  label_incident: "move to Expedite"
  merged_pull_request: "move to Verify"
  verified_production: "move to Done"

We now rotate a weekly triage owner. It takes about 35 minutes on a normal week and longer after an incident. The owner checks new work, pings stalled cards, and closes tickets that have become archaeology. Nobody loves the rota, but it is far less annoying than discovering during replenishment that Ready contains a request for Ubuntu 20.04 package support from last winter.

Our on-call rota deserves a brief mention here. We stopped making the on-call engineer triage owner after one week in September, when the same person had a noisy Kafka page at 02:17 and a backlog of 14 intake tickets by breakfast. The board can wait. A tired engineer should not have to decide whether a request is “standard” or “fixed date” before coffee.

We Kept Limits And Replenishment, Not Velocity Charts

The pieces we kept are the ones that force a decision.

The Doing limit is now six, not eight. We lowered it after measuring elapsed time and noticing that work got stuck at six anyway; the extra two slots were simply permission for two more partially started tasks. Six feels restrictive on Monday morning. By Thursday, it usually feels sensible.

We also kept twice-weekly replenishment. Tuesday and Friday, 20 minutes each, with the triage owner and whichever engineers are free. We pull enough Ready work to keep the board supplied, not enough to make everyone feel booked. We do not estimate every card. If a task is clearly larger than about five working days, we split it or write a short discovery card with a specific decision as its exit condition.

Our preference is blunt: we dislike story points for platform work. Giving an S3 lifecycle policy a five and a Kubernetes minor upgrade a thirteen produced elaborate certainty theatre. Elapsed time, blocked age, and a short list of current work tell us more.

We retained a monthly service-delivery review, although we cut the slide deck from 18 slides to four screenshots and a list of the oldest blocked items. We look at completion times by work type, not individual engineer. The point is to see that fixed-date work is clogging the queue or that access requests are incomplete, not to invent a ranking system for people who happened to draw the ugly tickets.

We have not tried this past 40 nodes or across multiple platform teams. Our cluster has 17 worker nodes and one team owns the board. At a larger scale, shared services and dependency boards may require a different arrangement. We are not claiming a magic ceiling; we simply do not have evidence beyond our own rather manageable mess.

We Ripped Out Swimlanes And Deadline Colours

We removed the swimlanes.

At one point the board had lanes for Kubernetes, CI, cloud accounts, observability, security, and “team enablement.” Cards duplicated across them, work vanished below the fold, and nobody could tell whether the WIP limit applied to the lane or the column. A board that needs a legend has already lost an argument with its users.

We also removed due-date colours. Red cards encouraged people to work on the nearest date even when the date was invented by a requester who wanted attention. Fixed-date work now has a documented external deadline and a reason. A request for “this sprint if possible” is standard work, even if the ticket title contains three exclamation marks.

The final thing we ripped out was the rule that every engineer must always have a card assigned. Pairing on a thorny rollout, reviewing a production verification, or helping the on-call engineer is work. It does not become more real because a board has somebody’s avatar attached.

One argument remains unresolved. Security remediation gets its own expedite slot when there is an active exploit, but we still disagree about high-severity findings with no known exploit. Half the team wants a second reserved slot. The rest of us think that would quietly become a permanent bypass around the limit. The next critical CVE will probably settle it for us.

Leadership Mistakes We Put Into Production Before Lunch

DevOps Oasis — Sat, 22 Aug 2026 05:40:47 +0000

Six management failures we shipped alongside perfectly valid infrastructure changes.

At 11:42 on Monday, the billing-export pull request had four approvals, one frustrated staff engineer, and no merge. Nothing in the diff was controversial. It changed a retry timeout from 30 seconds to 45. The problem was that our repository rules required a platform lead to approve anything touching infra/, and both platform leads were in a quarterly planning meeting.

We wrote that rule six months earlier. We then spent three hours discovering what it had actually done to the team.

We Made Leads The Default Merge Queue

What we did

We put @platform-leads in CODEOWNERS for most infrastructure paths. Any Terraform module, Kubernetes manifest, Helm chart, or GitHub Actions workflow waited for one of two people.

# Bad: .github/CODEOWNERS
/infra/              @platform-leads
/kubernetes/         @platform-leads
/.github/workflows/  @platform-leads

It looked orderly in the repository settings screen. We had senior review on high-impact changes. Nobody could accidentally modify production wiring on a quiet Friday afternoon. The fact that “quiet Friday afternoon” is when people make choices they regret did occur to us.

Why it looked reasonable

We had just hired three engineers who had not worked with our AWS accounts before. At the same time, a state-lock incident had left an S3 bucket policy too open for 47 minutes. We wanted a brake.

GitHub’s CODEOWNERS rules make this easy to set up, and easy to overuse. A broad path ownership rule feels like governance because it produces approvals. It doesn’t tell us whether the approval added useful scrutiny.

What it cost us

Between 4 and 15 August, 31 pull requests waited longer than two business hours for a required reviewer. Nine were harmless dependency bumps. Six were fixes to alerts already paging people. One was the retry timeout.

The less obvious cost was learned helplessness. Engineers started asking, “Can you just take this?” before they had read the plan output. Leads got worse at reviewing because our queue had become full of changes too small to deserve deep attention.

Our review time became an interrupt-driven service, which is a fairly miserable thing to run.

The fix

We moved ownership to the people who operate each service and kept lead review for a short list: identity, network boundaries, account creation, and production deletion paths.

# Corrected: .github/CODEOWNERS
/services/billing-export/  @billing-export-owners
/modules/retry-policy/     @platform-runtime
/modules/iam-boundary/     @platform-leads
/infra/accounts/           @platform-leads

We also added a risk:high label that requires one lead approval, rather than pretending every YAML file carries the same consequence. We pinned terraform 1.8.5 in .tool-versions on 2026-08-18, so reviewers were at least looking at plans produced by the same binary. Tool drift had been supplying enough fake disagreement already.

The new rule only held because our service owners already had production access and knew their own rollback paths. We have not tried it beyond 40 engineers or across a company with three separate compliance teams. We’d expect the ownership map to become its own small administrative animal there.

We Praised Firefighting In The Incident Channel

What we did

During the 03:18 orders-api incident on 7 August, a senior engineer found that an Envoy sidecar image had been pulled with a stale digest. They manually restarted six pods, traffic recovered, and the incident channel filled with grateful emoji.

Then we praised the restart in the weekly all-hands.

We did not praise the engineer for writing down the image-cache condition, because that happened two days later in a quiet document that fewer people read. We effectively taught the group that visible heroics counted more than removing the reason for heroics.

Why it looked reasonable

Incidents are tense, and people deserve credit when they act calmly under pressure. The restart did restore checkout. Nobody wants leaders to respond to a hard night with a lecture about process.

We also had a habit from smaller teams: the person who fixed the immediate problem was usually the person who could fix the whole problem. That stops being true when the system has 26 services, four cluster node pools, and a deployment controller that behaves differently at 03:00 than it did in staging. Or at least it feels that way after coffee.

What it cost us

Within ten days, the same stale-digest condition appeared in catalog-sync. The engineer on call restarted pods again because it was the known move. We lost 19 minutes of delayed catalogue updates and spent 14 person-hours on two nearly identical incident calls.

Worse, two people who had noticed the cache behaviour before the first incident did not bring it up in the review. They assumed an emergency workaround was the approved answer.

The fix

We changed our incident follow-up language. The incident commander now names three separate contributions: containment, diagnosis, and prevention. They are often done by different people.

Our postmortem template also includes “what made the fast workaround attractive?” That wording matters. “Why did we not fix it properly?” produces defensive writing and suspiciously clean timelines. The Google SRE postmortem guidance has the right instinct here: get the facts down without making the document a trial.

For the image issue, we set imagePullPolicy: Always for the affected deployment class, added digest validation to the release job, and created a runbook that says a restart is containment only. It is deliberately blunt.

If your team celebrates the person awake at 03:18, celebrate the person who makes 03:18 boring next month too.

We Assigned Pager Duty By Reporting Line

What we did

Our escalation policy routed alerts from payments, fulfilment, and customer-profile through the platform manager after the primary responder. The manager had enough context to coordinate work, so the routing felt sensible.

It also meant a people manager was the automated second-line responder for systems they did not deploy or change.

# Bad: escalation policy in Terraform
rule {
  escalation_delay_in_minutes = 10
  target {
    type = "user_reference"
    id   = pagerduty_user.platform_manager.id
  }
}

Why it looked reasonable

The platform manager attended architecture reviews, knew which teams were stretched, and could wake the right person. On paper, that resembles escalation.

We confused being informed with being useful at 02:00. Those are different jobs. One of them needs a laptop, current access, and enough familiarity with payment-webhook to distinguish a bad certificate from a bad deploy.

What it cost us

On 12 August at 01:56, a latency alert reached the manager after an on-call engineer was already investigating. The manager had to search the rota, message the service owner, and then relay a dashboard link. The service owner joined 17 minutes later.

No individual made a bad call. The policy encoded a slow handoff and asked tired people to compensate for it.

It also damaged trust in the rota. Service owners felt paged by proxy while platform absorbed blame for alerts outside its control.

The fix

We route to a service secondary first, then to an incident commander who can coordinate across teams. Managers get an informational notification after 20 minutes, not a wake-up call by default.

# Corrected: escalation policy in Terraform
rule {
  escalation_delay_in_minutes = 10
  target {
    type = "schedule_reference"
    id   = pagerduty_schedule.payments_secondary.id
  }
}

rule {
  escalation_delay_in_minutes = 20
  target {
    type = "schedule_reference"
    id   = pagerduty_schedule.incident_commander.id
  }
}

The PagerDuty escalation policy documentation is clear on how the mechanics work. The harder part is deciding who can act, rather than who sits highest on an org chart.

We still argue about whether managers should ever be in a daytime escalation chain. For customer-data exposure, probably yes. For a failed Kubernetes CronJob, absolutely not.

We Treated A Green Dashboard As Good Management

What we did

For most of July, our leadership update reported cluster availability, deployment completion, and ticket closure. All three were green. Meanwhile, engineers were bypassing the shared delivery path because preview environments took 38 minutes to appear.

Nobody hid that number. We simply had not made it a number we discussed with any seriousness.

Why it looked reasonable

Cluster availability was available in Grafana. Deployment completion came out of GitHub Actions. Ticket closure was in Jira, waiting politely for somebody to make a chart. These numbers were consistent and easy to report.

The preview-environment delay came from a mixture of runner queue time, Terraform state contention, and a slow image scan. It required actual investigation. We chose the numbers that arrived pre-chewed.

What it cost us

Two teams began keeping long-lived feature branches because they could not get useful feedback in a working day. One team deployed a feature flag directly to production to avoid another preview wait. That flag stayed enabled for five days because nobody was certain which branch had created it.

We had technically healthy infrastructure supporting increasingly unhealthy delivery habits.

The fix

We replaced the status slide with a weekly operating review of four measures: time from merge to production, failed deployment recovery time, on-call pages per service, and age of the oldest unowned alert. We use the definitions from DORA’s research program as a starting point, then adjust when the measurement lies.

For example, “merge to production” excludes changes waiting behind a scheduled customer release. Counting that delay as engineering delivery time would make a calendar meeting look like a platform defect.

We removed ticket closure entirely. We dislike it as a leadership metric. It rewards chopping work into small rectangles and then admiring the pile.

We Called On-Call A Growth Opportunity

What we did

When a product engineer joined the rota for the first time, we described it as a chance to learn the system. It was a chance to learn the system. It was also a week of interrupted sleep, a 06:40 database connection alert, and a Friday afternoon spent catching up on work they had planned to finish Tuesday.

We had no written rule for time back after a bad night.

Why it looked reasonable

Shared on-call builds ownership. We wanted product teams to see the consequences of their release decisions instead of handing every production concern to platform. That preference remains.

The mistake was treating the learning benefit as payment for the disruption. People do not need a motivational slogan after being woken by an alert caused by a threshold someone else set in 2023.

What it cost us

Three engineers swapped out of the rota in the next quarter. One did so quietly, citing “capacity.” We read that as a scheduling issue until their manager told us they had been paged five times in two weeks and had received no protected recovery time.

Alert quality also got worse because responders deferred cleanup work. Nobody volunteers to improve an alert that might later wake them if the work competes with a sprint commitment.

The fix

A page between 23:00 and 06:00 now earns a late start or half-day recovery without negotiation. A second page in the same night triggers a manager check-in and a review of the alert within five working days.

We also publish the rota alongside expected service work, so a person carrying orders-api is not assigned a migration cutover that week. This costs delivery capacity. Good. The capacity was fictional before; it was being borrowed from somebody’s sleep.

We Replaced Decisions With Status Meetings

What we did

Our Thursday platform meeting had 14 people, 45 minutes, and a standing agenda full of updates. The recurring item was “progress on secrets migration.” For six weeks, everyone reported progress while nobody decided whether service teams had to move by the September deadline.

The migration repository had 63 open issues. Twelve were blocked on choices that no ticket owner could make.

Why it looked reasonable

Status meetings make managers feel informed. They also give people a place to raise concerns, which sounds generous until the same concern appears in six consecutive agendas with no owner assigned to resolve it.

We had been trying to avoid unilateral decisions after a previous migration upset two application teams. Consensus became the safer social move. It was also much slower.

What it cost us

By the time we set a policy, the legacy Vault token format had spread to two new services. The team then had to support both formats during the busiest release month of the quarter.

The meeting itself cost 31 person-hours over those six weeks. The delay cost more, but the meeting cost was easier to calculate and therefore more annoying.

The fix

Every recurring operational meeting now has a decision log with three fields: decision required, named decider, and deadline. Updates go in writing before the meeting. If no decision is pending, people get 45 minutes back.

For the secrets migration, we chose a hard date: new services after 2026-09-15 must use workload identity; existing services get a supported adapter until 2026-12-31. We accepted that two teams would dislike the date more than they disliked ambiguity.

The adapter is still ugly. We have not found a clean migration path for the vendor service that only accepts a static credential, and nobody is excited about the options.

SRE Starts With The Alert That Woke Us

DevOps Oasis — Sat, 22 Aug 2026 05:40:26 +0000

How we made service reliability a team job, not an on-call punishment.

The payment-api alert was still firing when we opened the incident channel, and nobody could tell whether customers were actually blocked or whether a dashboard had found another way to ruin breakfast.

SRE means site reliability engineering: a way of running software where the people who build services also set clear expectations for how those services should behave in real use. We measure whether they meet those expectations, respond when they do not, and spend planned time removing repeated sources of failure.

At 08:06, we had a red graph, four Slack replies, and no useful answer to the only question that mattered: could a customer complete a payment? That gap is why we started treating SRE as a working practice rather than a title somebody receives after learning enough Kubernetes nouns.

Start With What A User Can Actually Do

A service can have healthy pods, low CPU use, and a very cheerful deployment status while still failing the one task people came to perform. We learned this with payment-api in April. Its /healthz endpoint returned 200, every container was running, and the PostgreSQL connection pool looked normal. Meanwhile, requests to create a payment were waiting 18 seconds on a call to our fraud provider.

Nobody buying a train ticket cares that our health endpoint is fine.

For SRE work, we begin with a user action: create an account, load a report, upload a document, complete a payment. That action gives us something concrete to measure and protect. It also stops teams from treating infrastructure metrics as proof that an application works.

We wrote down three actions for the payments service:

A customer can submit a payment.
A merchant can retrieve its payment status.
Our nightly settlement worker can send a batch to the bank before 02:00.

Those are not equally valuable. A delayed merchant status page is annoying. A payment request that disappears or hangs is a financial incident. Settlement missing its bank window has a different kind of unpleasantness, usually involving spreadsheets and a person from finance who has every right to be annoyed.

The Google SRE book’s section on service level objectives is useful here because it separates what users experience from whatever machinery happens to sit behind it. We borrowed the idea, then made the wording much plainer in our own service docs.

A user action is the unit we protect; the rest is evidence.

Measure Requests, Not The Mood Of The Cluster

Once we had a useful user action, we needed a number. For payment submission, we count successful HTTP requests and failed HTTP requests. A request is successful if it returns a 2xx response within five seconds. A 500, a timeout, or a response after five seconds counts as bad.

That five-second line is not sacred. We picked it after looking at 30 days of request timing. Most payments completed in 430–900 ms. At 2.5 seconds, support tickets started to correlate with retries. At five seconds, our web client gave up and showed the customer an error. We chose the point where the customer’s experience had already become poor, rather than pretending 400 ms was a promise we could keep during the Monday morning rush.

In SRE terms, the thing measured is a service level indicator, usually shortened to SLI. We call it “the payment success rate” in conversation because acronyms spread quickly enough without our help.

This Prometheus query is close to the one on our dashboard:

sum(rate(http_requests_total{service="payment-api",code=~"2.."}[5m]))
/
sum(rate(http_requests_total{service="payment-api"}[5m]))

The query is deliberately boring. It counts good requests divided by all requests. It does not estimate customer happiness from node load or count pods with names containing payment. Those metrics still have a place when diagnosing a fault, but they are poor promises to make to users.

We also exclude requests made by our synthetic checker from this ratio. Otherwise a healthy checker can keep the graph looking better while ordinary traffic is failing, which is exactly the sort of technical correctness we dislike.

A useful SLI maps a visible user action to a number we can inspect during an incident.

Set A Target You Can Defend In A Meeting

A service level objective, or SLO, is the target for that measurement over a period of time. Ours for payment submission is 99.9% successful requests in a rolling 28-day window.

That means we permit roughly 43 minutes of failed or slow request time per month if every request fails at once. Real faults are messier than that: a partial outage may consume the allowance across thousands of requests in ten minutes. Still, the number gives us a shared boundary.

We tried 99.99% first.

It did not work.

The target sounded suitably serious in the planning document, but it gave payment-api about four minutes and 19 seconds of failure allowance in 30 days. A routine database failover consumed most of that. A deploy that caused a two-minute spike put us in a state where the sensible response was “stop changing anything,” including a fix for the thing that had just failed. We changed it after two months. Nobody has suggested going back.

An SLO should reflect the cost of failure and the cost of making change. For payment submission, 99.9% is defensible because a short failure is costly, but we also deploy fixes and dependency changes every week. For our internal analytics export, 99.5% is plenty. The report is useful at 09:00; it does not need a ceremonial response at 03:00 because one CSV was 12 minutes late.

The target is a negotiated engineering decision, not a number copied from a cloud provider’s brochure.

Treat The Error Budget As Permission To Change

The difference between the target and actual performance is called an error budget. If our target is 99.9%, the remaining 0.1% is the amount of bad service we have agreed can happen in the measurement window.

That phrasing can sound grim. We prefer to see it as permission to take normal engineering risks. Deployments, schema changes, cache migrations, and dependency upgrades all carry some chance of disruption. If we demand zero disruption forever, we get slower changes, stale dependencies, and people quietly avoiding work that needs doing.

At 11:35, during the payment incident, the budget had fallen from 71% remaining to 63%. That gave us room to ship a timeout change and route a small percentage of traffic around the fraud provider. Had we only had 2% left, we would have made a different call: freeze feature releases, use the safer rollback path, and put the reliability work at the top of the sprint.

We keep the policy small:

payment-api:
  slo: 99.9
  window: 28d
  budget_remaining_below: 10%
  release_action: "approval from on-call and service owner"

This is not an automatic deployment lock. We tried a hard lock in GitHub Actions last year, and engineers worked around it by classifying urgent changes as operational patches. The YAML survived; the lock did not.

An error budget gives release decisions a number to argue from instead of relying on whichever person is most worried in the call.

Page People Only When They Can Act

An alert should tell an awake person that they need to do something now. Not investigate later. Not admire a graph. Not acknowledge it so that the alert noise reduces by one.

Our old alert set had 146 rules for a cluster of 24 nodes. At 02:40 one Thursday, the on-call engineer received 17 pages for a single Redis failure: node disk pressure, pod restarts, connection errors, cache miss rate, eviction count, and several alerts that had apparently been written by a committee of anxious refrigerators.

We cut that down. The page now fires when the payment SLO is burning quickly enough that we are likely to consume 5% of the monthly budget in one hour. The disk and restart alerts still exist, but they go to a working-hours channel unless they are directly causing user failure.

This uses the “burn rate” approach described in the SRE workbook’s alerting chapter. We like it because it joins urgency to impact. A five-minute blip at 03:00 can wait. A failure eating a month’s allowance before lunch cannot.

The alert notification includes three links: the SLO graph, the relevant deployment history, and the runbook. It does not contain fourteen labels, a container hash, and the hostname of a node that may no longer exist by the time somebody reads it.

The runbook is where we put immediate checks, known failure modes, and safe rollback steps. It is not a wiki page titled “Payments Operational Knowledge” last edited in 2022 by someone who moved teams.

A page earns its place on the rota only when an engineer has a real action available.

Reliability Work Has To Compete For Sprint Time

SRE fails when reliability tasks become “we’ll get to it after the roadmap.” There is always another roadmap item. There is also always another slow query, expiring certificate, or queue consumer that only fails after a long weekend.

We reserve capacity in each service team’s sprint for reliability work. Not a fixed 20% in every case; fixed percentages create odd incentives and fake tickets. Instead, the service owner brings the largest repeat incident, highest-risk dependency, or most worrying budget trend to planning. We ask what would make that problem less likely next month.

For payments, the answer this quarter was not another dashboard. We had plenty of dashboards. We added a circuit breaker around the fraud provider, recorded dependency timing separately, and changed the retry policy so that 400 failed requests do not all retry in the same second.

Our colleague Mina still thinks circuit breakers are a polite name for “giving up early.” She is partly right. We are still arguing about whether failed fraud checks should enter a manual review queue during a provider outage or be rejected immediately. That decision crosses risk, support staffing, and customer trust; no latency graph will settle it for us.

We track reliability work in the same backlog as product work. If it needs a separate spreadsheet to exist, it is already losing.

Words We Use Before They Become Meeting Noise

A short glossary helps because SRE has enough abbreviations to make ordinary incident calls sound like a radio test.

Service level indicator (SLI): The measurement of a user-facing action. For us, payment requests completed successfully within five seconds.

Service level objective (SLO): The target for that measurement over a time window. Our payment submission target is 99.9% over 28 days.

Error budget: The allowed amount of failure implied by the SLO. It is what remains when actual performance is compared with the target.

Burn rate: How quickly the service is spending its error budget. A fast burn rate deserves attention even if the monthly number still looks acceptable.

On-call: The person scheduled to respond when an alert pages. This should rotate, include proper handover, and not quietly become one person’s permanent second job.

Runbook: A practical page for responding to a known alert: where to look, what is safe to change, and when to escalate.

Prometheus has a clear alerting rules reference if you are building alerts from scratch, while Alertmanager’s grouping documentation explains how to avoid receiving seventeen versions of the same bad news.

Words are useful only if the person holding the pager can turn them into a decision.

Keep The First Version Small Enough To Maintain

We have not tried this approach past 40 nodes or across dozens of independently deployed teams. At our size, one platform group and six service teams can still agree on common labels, a dashboard layout, and who owns an alert. Larger organisations will need more structure, and probably more patience than we had at 08:06.

Start with one service that matters, one user action, one measured target, and one page that somebody can act on. Do not wait for a grand reliability programme. We began with payment submission because its failures were visible, expensive, and already waking people up.

By 16:40, the fraud provider had recovered, the error budget had stopped falling, and the incident channel had gone quiet. We still have a node_filesystem_avail_bytes alert that wakes people too often on a particular worker pool. It has been open for 19 days, which is annoyingly longer than the incident that made us write this post.

Ansible Myths We Met During The Debian 12 Cutover

DevOps Oasis — Sat, 22 Aug 2026 05:40:06 +0000

What held up across 28 hosts, and what absolutely did not.

02:13.

That was when the last Debian 11 VM in our payments group stopped accepting the old OpenSSH cipher list, halfway through an Ansible run that had looked boring for 41 minutes. Boring is usually a compliment in infrastructure work. At 02:13, it became a theory.

We moved 28 application and utility hosts to Debian 12 over three evenings: 14 API nodes, six Celery workers, four reporting boxes, two Redis replicas, and two machines nobody could name without checking NetBox. We ran ansible-core 2.17.7 from a pinned Python 3.12 virtual environment on our control host. The exercise produced a familiar set of opinions from people who have used Ansible once, inherited it reluctantly, or seen a YAML file commit a minor crime.

Most of those opinions contain a grain of truth. Grains are how people end up eating sand.

“Ansible Is Just SSH With YAML”

Verdict: false, though SSH is still where many bad evenings begin.

For ordinary Linux hosts, Ansible’s default transport is SSH, and that fact encourages people to dismiss the rest as a collection of remote shell commands with more indentation. We did use SSH. We also used inventory groups, facts, idempotent modules, handlers, privilege escalation rules, check mode, variable precedence, and a run history we could inspect after someone asked why cron had restarted on a reporting node at 01:47.

Calling all of that “just SSH” is like calling PostgreSQL “just files on disk.” Technically defensible, professionally unhelpful.

The distinction mattered during the cutover because our hosts were not alike. The API nodes run gunicorn behind Nginx. Celery workers have different systemd limits. The reporting boxes mount an NFS share that the application hosts never see. A shell loop would have needed its own logic for each difference, along with error handling and a way to tell whether a retry had changed anything. We have written those loops before. They age like yoghurt in a server room.

Here is the small part of the play that put the right APT source on each Debian 12 host and notified Nginx only when its file changed:

- name: Configure internal package source
  ansible.builtin.deb822_repository:
    name: oasis-internal
    types: [deb]
    uris: "https://apt.devopsoasis.internal/debian"
    suites: ["{{ ansible_distribution_release }}"]
    components: [main]
    signed_by: "https://apt.devopsoasis.internal/keys/archive.gpg"
    state: present
  notify: Reload nginx

- name: Install application packages
  ansible.builtin.apt:
    name:
      - oasis-api={{ api_package_version }}
      - nginx
    state: present
    update_cache: true

handlers:
  - name: Reload nginx
    ansible.builtin.service:
      name: nginx
      state: reloaded

The Debian repository module and the APT module gave us useful state instead of an optimistic curl | tee sequence. We could rerun the play after fixing the cipher issue and see 23 hosts report ok, four make the expected package changes, and one fail on a stale internal repository key.

That said, Ansible does not remove SSH’s sharp edges. A stale known_hosts entry, a host key rotation, an expired jump-host certificate, or a control socket left under /tmp can still stop a run before a task begins. We had all four categories represented in the preceding month, because apparently our infrastructure enjoys variety.

Ansible gives SSH a memory and a grammar. It does not grant SSH better manners.

“If A Playbook Is Idempotent, Rerunning It Is Safe”

Verdict: dangerously false.

Idempotence means a task should reach the same desired state when applied repeatedly. It does not mean every action surrounding that task is harmless. A package install can be idempotent while the package’s post-install script restarts a service. A template can be idempotent while its handler drains a connection pool at a bad moment. A database migration can be “already applied” and still leave a node with an incompatible binary if deployment order goes wrong.

We had a clean example on the Celery workers. The role installed oasis-worker, rendered a systemd unit, and restarted the service when either changed. On Debian 11, the worker used a unit file with Restart=always. On Debian 12, we changed it to Restart=on-failure after discovering that a bad RabbitMQ credential caused a very energetic restart loop and 9 GB of logs over an on-call shift.

The first run changed the unit file. Good. The second run should have reported no changes. Instead, it restarted all six workers because a task used state: restarted as a convenient substitute for thinking.

We replaced it with this:

- name: Install worker systemd unit
  ansible.builtin.template:
    src: oasis-worker.service.j2
    dest: /etc/systemd/system/oasis-worker.service
    owner: root
    group: root
    mode: "0644"
  notify:
    - Reload systemd
    - Restart oasis worker

- name: Ensure worker is enabled and running
  ansible.builtin.systemd_service:
    name: oasis-worker
    enabled: true
    state: started

handlers:
  - name: Reload systemd
    ansible.builtin.systemd_service:
      daemon_reload: true

  - name: Restart oasis worker
    ansible.builtin.systemd_service:
      name: oasis-worker
      state: restarted

The handler behaviour documented by Ansible is useful here: notifications coalesce, so six changed tasks still cause one restart at the end of a play. That only helps if we notify on real changes and avoid casually restarting a process because the playbook happened to visit it.

There is a second trap: changed_when: false. People add it to quiet noisy shell tasks, then forget that they have hidden the only signal a downstream handler had. We found one role doing exactly that around update-ca-certificates. It was marked unchanged even when it had imported a new internal CA. The application unit therefore kept running with an old trust store until its next scheduled restart.

Silence in Ansible output is not proof that nothing happened.

We now treat reruns differently by class of work. Package and file convergence can run repeatedly. Service reloads need a reason. Schema changes get a separate deployment step and a named human watching the output. We still have an argument going about whether certificate rotation belongs in the same play as application rollout. Marta, who carries the Thursday primary rota, wants it split permanently. We have not settled it, mostly because nobody enjoys owning the extra release button.

“Roles Make Every Ansible Repository Easier To Maintain”

Verdict: false after the third layer of includes.

Roles are useful. We keep roles for things with a real lifecycle: oasis_nginx, oasis_api, oasis_worker, node_exporter, and vector_agent. Each has defaults, templates, handlers, and tests. That structure lets a new engineer find the file that owns /etc/nginx/sites-enabled/oasis-api.conf without hiring a tracking dog.

Then roles become a way to hide choices.

Our oldest repository had a common role that installed base packages, set SSH settings, configured journald, added users, installed monitoring, patched APT sources, mounted volumes, and once, for reasons lost to history, configured mailutils. Every host received it. Its defaults file had 67 variables. The role’s README said “foundation configuration,” which meant nobody knew whether changing it would restart half the fleet.

We removed common during the Debian 12 work. Not renamed it. Removed it.

The replacement is less elegant to people who enjoy directory trees. The base play now names what it does:

roles:
  - role: base_packages
  - role: ssh_hardening
  - role: journald
  - role: node_exporter
  - role: vector_agent

That is more repetitive, and we prefer it. A deployment plan should make its dependencies visible. When ssh_hardening changes, we can run it against the staging bastion first rather than wonder what else common believes it owns this week.

We also stopped putting application-specific variables in group_vars/all.yml. That file had become the cupboard under the stairs: Redis memory limits beside deploy keys, Nginx rate settings beside a legacy hostname, each one apparently too small to deserve a proper home. Variable precedence in Ansible is powerful enough to support almost any arrangement, which is a polite way of saying it will let us build a difficult mess. The precedence rules are clear; the harder part is declining clever arrangements before they spread.

Roles help when they represent a bounded unit of ownership. A role that changes SSH, packages, users, metrics, and mail is a landfill with a defaults/main.yml.

“Check Mode Tells Us What Production Will Do”

Verdict: partly true, and we still run it on every merge.

ansible-playbook --check --diff catches a surprising amount. It spots malformed templates, missing variables, unexpected file diffs, and package changes that would otherwise appear during a production window. For the Debian 12 migration, check mode caught an Nginx template referring to ssl_protocols TLSv1.2 TLSv1.3; on a host group where TLS terminated at the load balancer. The task was valid. The configuration was unnecessary. That distinction saved us a small amount of clutter, which is the usual reward for doing review properly.

It cannot predict everything.

Modules vary in their check-mode support. Commands and scripts cannot know their impact unless we tell Ansible with creates, removes, or explicit changed_when logic. External systems are worse. Our deployment task calls an internal release API to register the package version. In check mode, we skip it. Running a fake registration against production would turn a dry run into an incident rehearsal.

Package managers also have opinions. APT can calculate planned installs, but the final answer depends on repository contents, dependency resolution, held packages, and whether the mirror has finished syncing. On the second cutover night, our internal mirror had libpq5 15.8 for eleven minutes before its metadata caught up. Check mode at 22:06 looked fine; the real run at 22:17 installed nothing until we forced an index refresh.

This only held because our package mirror is local and the sync lag is usually under 15 minutes. We have not tried this process across 40 nodes or across regions with separate mirrors. The current serial batch size of two works because someone can read the output before the next pair begins, not because two is a mystical number.

Check mode is a review aid. Treating it as a production simulation invites a very confident failure.

“Ansible Is Too Slow For Real Deployments”

Verdict: mostly true, if “real deployments” means every host deserves parallel attention.

Ansible is not slow in the way people mean when they complain about it. A play spends time connecting, gathering facts, copying files, waiting for package locks, and asking remote services whether they already match the requested state. That work is visible. A bash fan-out script hides the same waiting behind interleaved output and gives it a name like deploy-final-v4.sh.

For 28 hosts, our full Debian 12 convergence play took 18 minutes and 36 seconds with forks = 10. Fact gathering alone consumed just under four minutes on the first run. We could turn it off, and we did for the application-only deployment play. We did not turn it off for the OS migration because several templates depend on distribution release and interface facts, and discovering that late is tedious.

The larger problem is that Ansible’s default execution model makes safe rolling work feel deliberate. We set serial: 2 on API nodes, wait for each host to return to the load balancer, and only then move on. That is slower than replacing all 14 at once. We prefer slower. The incident report from 2023, when a bad Nginx include removed health-check responses across an entire pool, remains available to anyone tempted by speed.

We do use the free strategy for low-risk work such as updating node exporter and Vector. It prevents a slow reporting host from making every other machine wait at the same task boundary. We do not use it for application releases, where matching order across a small batch is easier to observe and roll back.

Ansible has a ceiling. We would not choose it to coordinate thousands of ephemeral containers, and we dislike pretending a tool should solve jobs it was not built for. For a few dozen long-lived servers with awkward local differences, the minutes it spends checking state are cheaper than the minutes we spend explaining a broad outage.

Jenkins Versus GitHub Actions After Our 43-Minute Queue

DevOps Oasis — Sat, 22 Aug 2026 05:39:45 +0000

What we kept, what we moved, and why Jenkins still runs one awkward job.

At 10:17 on Tuesday, our release branch had 19 builds waiting behind a Jenkins controller that was technically healthy and practically useless. The oldest job had been queued for 43 minutes. Two engineers were watching the same green progress bar on Slack. One of them suggested “just adding executors,” which is how Jenkins makes sure we remember it has a sense of humour.

We compared three paths for the services we build and deploy from GitHub: keep Jenkins on Kubernetes, move normal CI work to GitHub Actions, or run GitLab CI for the teams already using GitLab’s package and security features. This wasn’t a feature checklist exercise. We had 34 repositories, roughly 280 builds on a busy weekday, Java and Node services, Terraform plans, and one antique deployment job that talks to a vendor appliance through a jump host.

Our plain answer: GitHub Actions won for most application CI. Jenkins stayed for the strange jobs with network access, custom hardware, or years of scripted behaviour nobody wants to translate on a Thursday afternoon. GitLab CI is a decent choice if GitLab already owns your source control. We would not introduce it beside GitHub merely to replace Jenkins.

The Queue Problem Is Really a Worker Problem

Our Jenkins controller ran as a StatefulSet in EKS, with ephemeral Kubernetes agents created through the Jenkins Kubernetes plugin. On paper, this gave us elastic executors. In practice, the agent pod templates had grown into a 480-line shared library, node images were pulled too often, and a Docker-in-Docker sidecar occasionally sat in ContainerCreating while its sibling job waited.

The controller itself wasn’t the bottleneck. The queue was.

We had capped the ci-build node group at 12 m6i.large instances because it already cost enough during normal hours. Each Maven build requested 2 CPU and 4 GiB RAM. A few test suites asked for 6 GiB, Kubernetes packed them poorly, and cluster autoscaler took four to seven minutes to add nodes. By the time capacity appeared, another branch push had arrived.

GitHub Actions moved that waiting time somewhere else. Hosted runners started most Node and unit-test jobs in under a minute. The Windows jobs were less charming, but we only have four of those. GitLab’s shared runners behave similarly, though we have seen queue time vary more with GitLab.com’s shared capacity than we like for release work.

Here’s the blunt trade-off:

Criterion	Jenkins on Kubernetes	GitHub Actions	GitLab CI
Normal app build startup	2–8 minutes in our cluster	15–60 seconds	30–120 seconds on shared runners
Custom network access	Excellent	Requires self-hosted runner	Requires self-managed runner
Pipeline code	Powerful, often overgrown Groovy	YAML plus actions	YAML, generally cleaner than Jenkinsfiles
Plugin and action risk	Plugin upgrades can break controllers	Third-party actions need pinning	Templates and includes need review
Cost shape	EKS nodes, storage, admin time	Per-minute after included use	Per-minute or runner infrastructure
Debugging odd failures	Logs spread across controller, pod, cluster	Usually one run page	Usually one pipeline page
Our recommendation	Keep for exceptional jobs	Default for GitHub-hosted teams	Default only for GitLab-hosted teams

The surprise was that cost got worse before it got better. In the first month after moving 22 repositories, our GitHub Actions bill rose from about $180 in trial usage to $1,460. We had copied Jenkins behaviour too literally: every pull request ran integration tests, image builds, dependency scans, and a 19-minute Playwright suite.

Then we split checks by path, cancelled superseded runs, and put browser tests behind a merge-queue label. The next month landed at $690. Jenkins node spend also dropped by roughly $510, though our finance report made that connection about as obvious as a failed Helm rollback.

The Same Maven Build In Three Pipeline Files

We used the same service for the comparison: billing-api, a Spring Boot service with Maven tests, a container image, and a Helm deployment to staging. It has enough moving parts to expose weak spots, but it isn’t one of the cursed repositories.

Jenkins still gives us the most freedom. It also gives every team enough rope to build a second deployment system inside a shared library.

pipeline {
  agent {
    kubernetes {
      yaml """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: maven
    image: maven:3.9.9-eclipse-temurin-21
    command: ['cat']
    tty: true
"""
    }
  }

  stages {
    stage('Test') {
      steps {
        container('maven') {
          sh 'mvn -B -DskipITs verify'
        }
      }
    }
    stage('Image') {
      when { branch 'main' }
      steps {
        sh './ci/build-and-push.sh'
      }
    }
  }
}

The GitHub Actions version is shorter, and more of its behaviour is visible to a developer who has never had Jenkins administrator access. We pin third-party actions to commit SHAs in the real files; abbreviated tags below are easier to read but not what we permit in protected branches. GitHub’s own security hardening guidance is sensible here, particularly on action pinning and token permissions.

name: billing-api

on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read
  packages: write

jobs:
  test:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: "21"
          cache: maven
      - run: mvn -B -DskipITs verify

  image:
    needs: test
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4
      - run: ./ci/build-and-push.sh

GitLab CI is the least surprising YAML of the three for a conventional pipeline. Its rules syntax is better than the conditional knots we’ve seen in Jenkinsfile Groovy. GitLab’s runner executor documentation is worth reading before committing to shared versus Kubernetes runners; the operational split matters more than the syntax.

stages:
  - test
  - image

test:
  image: maven:3.9.9-eclipse-temurin-21
  stage: test
  cache:
    paths:
      - .m2/repository
  script:
    - mvn -B -DskipITs verify

image:
  stage: image
  image: docker:27
  services:
    - docker:27-dind
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
  script:
    - ./ci/build-and-push.sh

For a plain service, we’d choose Actions. Jenkins’ extra power only pays for itself when the job really needs it, not when someone wants an input step and a decorative Build With Parameters page.

Credentials Are Where Jenkins Shows Its Age

We spent more time on credentials than migration scripts.

Jenkins had 67 credentials in its store when we started. Twelve were still in use. Seven had names like prod-token-final2, which conveyed neither ownership nor comfort. A folder-scoped credential can be perfectly reasonable, but a Jenkins administrator can still reach a great deal of it, and plugin-specific credential bindings make audits tedious.

For GitHub Actions, we switched AWS deployments to OIDC and removed long-lived access keys from the pipeline path. The trust policy limits the role to our platform/billing-api repository and the main branch environment.

{
  "Condition": {
    "StringEquals": {
      "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
      "token.actions.githubusercontent.com:sub":
        "repo:acme-platform/billing-api:environment:production"
    }
  }
}

That was one of the few changes that felt immediately cleaner. The deploy job requests a short-lived token, AWS records the role session, and we stop pretending a secret rotated every 90 days is a happy secret. GitHub documents the OIDC flow for AWS well enough that we didn’t have to invent a wrapper.

GitLab supports the same general pattern with ID tokens. If your repositories live there, use them. We would avoid storing cloud keys in either CI product unless the target system has no federation option.

Jenkins can do OIDC too, via scripts, plugins, or an external secrets operator. We have it working for two jobs. The issue is consistency: every Jenkins library can choose its own route, and then the security review turns into archaeology.

We left one Jenkins credential in place: a client certificate required by that vendor appliance. Its API only accepts mutual TLS from a fixed private address, and its certificate rotation procedure arrives as a PDF once a year. Some systems do make their own case for retirement.

Could we move that appliance job to a self-hosted GitHub runner and shut Jenkins off entirely?

Self-Hosted Runners Recreate Part of the Jenkins Problem

Yes, we could. We tested it.

A self-hosted Actions runner in the private subnet could reach the appliance, mount the certificate from Secrets Manager, and execute the deployment script. The workflow itself was fine. The trouble began when we considered who would patch, drain, scale, and investigate those runners at 02:00.

Jenkins already had that machinery, imperfectly. Our agents were Kubernetes pods, isolated per build, and constrained with service accounts and network policies. A self-hosted runner can be ephemeral too, and GitHub recommends ephemeral runners for security-sensitive work. But we would still need an autoscaler, images, AMI or container patching, labels, runner registration, and enough logging to answer “what ran on this host?”

That’s Jenkins-shaped work wearing a GitHub badge.

We tried Actions Runner Controller for a week on a non-production cluster. It worked, but it added another controller, runner scale sets, GitHub App credentials, and operational ownership. Our platform team has five people, one of whom is on parental leave until October. We don’t have spare enthusiasm for another control plane.

This only held because the appliance job runs two or three times a week. If it ran 200 times a day, we would build the runner pool and accept the maintenance. At that volume, manual exceptions become their own incident category.

For now, Jenkins owns the private-network and hardware-adjacent jobs: appliance deploys, Android signing, and a load-test suite that pushes 25 Gbit/s through a lab VLAN. There are nine of them. Everything else is moving out.

Failed Builds Need Fewer Places To Look

Jenkins failures trained our team to open three tabs: build log, Kubernetes events, and Grafana. Four tabs if the shared library had changed. A red Jenkins job might mean a test failed, an agent could not schedule, the container registry throttled us, or the controller had lost a WebSocket connection to an agent.

GitHub Actions collapses most normal failures into one place. Logs are still noisy, especially when an action emits coloured furniture instead of useful output, but the run page has the checkout, cache, test, artifact, and deployment evidence together. GitLab offers a similar experience.

There’s a cost to that convenience. Hosted runners hide the host. When a GitHub ubuntu-24.04 image changed and our Playwright browser install broke, we had less ability to inspect the runner than we had with our own pods. Pinning the browser package fixed it, but the first failure message was just:

Error: browserType.launch: Executable doesn't exist

Our Jenkins setup had a different failure mode: the browser image was pinned for so long that it contained a six-month-old Chromium build. Predictability can quietly turn into neglect.

We also found that GitHub’s concurrency control stopped a lot of wasted work:

concurrency:
  group: pr-${{ github.event.pull_request.number }}
  cancel-in-progress: true

A developer pushing five commits in ten minutes no longer buys five copies of the same integration run. Jenkins can do this with milestone steps, abort plugins, or custom Groovy. We had all three patterns across repositories, naturally.

Marta, our Tuesday primary, preferred Jenkins logs because “at least the disaster is familiar.” She was right for the first fortnight. By week three, she stopped asking which agent pod had vanished.

Plugin Freedom Has a Maintenance Bill

Our Jenkins controller had 46 installed plugins. We removed 11 before the migration and still found dependencies on workflow APIs nobody had consciously selected. A plugin update is rarely exciting until it breaks a shared library and 34 repositories lose the ability to publish test reports.

We held Jenkins core at 2.452.3 for longer than we should have because an upgrade changed behaviour in the Kubernetes plugin. That wasn’t irresponsible exactly; it was a calculated refusal to spend a Friday proving that 120 pipelines still behaved. The calculation gets less defensible each quarter.

GitHub Actions has its own supply-chain problem. Marketplace actions are code from strangers with a reassuring logo. We allow approved actions, pin them by SHA, and use reusable workflows for container publishing and deployment. Teams can read those workflows, but they cannot casually add curl | bash to production release paths.

GitLab’s includes and templates offer a similar central-control model. We prefer it over Jenkins shared libraries because YAML is less capable of becoming a small programming language with opinions about classpaths. Some people love Jenkins Groovy. We have enough Java already.

There remains one unresolved argument in our team: whether reusable GitHub workflows are becoming shared libraries with worse local testing. They probably are. We’ve agreed to keep them thin—authentication, artifact naming, policy checks—and leave build commands in each repository. Ask us again after the next 20 migrations.

Pick This If Your Team Has These Constraints

Pick GitHub Actions if you have 3–30 engineers, repositories already hosted on GitHub, and mostly standard builds: tests, containers, Terraform plans, and cloud deployments. Start on hosted runners. Use OIDC for cloud access, protect environments, cancel duplicate pull-request runs, and resist adding self-hosted runners for one awkward service.

Pick Jenkins on Kubernetes if you have 15–80 engineers and a meaningful set of jobs that require private network paths, internal build hardware, custom operating systems, or long-established scripted release steps. Keep the controller small, make agents ephemeral, and assign ownership for every shared library. Do not keep Jenkins merely because the team knows where the blue button is.

Pick GitLab CI if source code, merge requests, container registry, and security scanning already live in GitLab. For a 10–50 person team, one product for those workflows is a sensible operational trade. We would choose GitLab CI over Jenkins in that setup, but not over GitHub Actions if moving source control is the hidden price.

For our five-person platform team, the split is 25 repositories on GitHub Actions, nine Jenkins jobs left behind, and a calendar reminder in November to see whether that vendor appliance has finally learned what an API is.