DEV Community: DevOps Oasis

DevOps Questions After We Broke The Release Handshake

DevOps Oasis — Sat, 22 Aug 2026 06:07:26 +0000

Answers from the incident where every dashboard looked politely wrong.

The release-api deployment had already been marked complete when the invoice page began returning 503s. The new container was serving traffic, the PostgreSQL migration had committed, and the feature flag was on. A NetworkPolicy added in another repository prevented the new pod from reaching tax-rate-cache.

The application team saw errors, the database team saw a clean migration, and Platform saw green nodes. By the time we put all three facts in one incident channel, 63 deployment messages had buried the one that mattered.

“Is This Actually A DevOps Failure Or Just One Bad Deploy?”

It was a DevOps failure because four teams completed valid local work and nobody owned the release handoff between them.

Calling it “just a bad deploy” would have been convenient. We could have fixed the policy, replayed the release, written a short incident note, and carried on pretending that a green Argo CD application means a service is ready for users. The pod was healthy according to Kubernetes. It was also unable to call a dependency required to render an invoice. Both things can be true, which is why a deployment status alone is a fairly poor witness.

Our old release process had hidden contracts in too many places:

The service repository declared its image and Helm values.
The infrastructure repository held network rules.
Database migrations ran from a separate GitHub Actions workflow.
Feature flags lived in LaunchDarkly, owned by whoever had last touched the feature.
The runbook lived in Confluence, where it had last been edited in February.

We’ve started putting the release dependencies in the service repository, close to the code that needs them. It is not a clever system. It is a file that a human can read during an incident and a pipeline can check before promotion.

release:
  service: release-api
  requires:
    - dependency: tax-rate-cache
      namespace: finance
      port: 8080
      network_policy: allow-release-api-to-tax-rate-cache
    - migration: 2026_08_17_add_tax_region
      minimum_version: "4.12.0"
  feature_flags:
    - invoice_tax_v2
  smoke_test:
    command: "./scripts/check-invoice-preview.sh"
    timeout_seconds: 90

The pipeline does not yet validate every field. It checks that the named NetworkPolicy exists in the target cluster and runs the smoke test after the migration. That alone would have caught Monday’s problem before 11% of invoice-page requests failed.

We dislike using “DevOps” as a department name. In this case, it means the work of making handoffs visible before production discovers them for us.

“Do Developers Really Need Production Access?”

Yes, but they need narrow production access tied to the service they own, not a shared kubectl credential passed around like an office spare key.

For too long, our answer was to give engineers log access and tell them to ask Platform for anything else. That worked while we had six services and one cluster. We now have 34 services, three production clusters, and an on-call rota where the person holding the pager may know the application code better than the person who built the cluster.

The useful access is usually boring:

Read logs, events, and deployment state in the team namespace.
View configured secrets by name, never by value.
Restart a failed deployment through a recorded workflow.
Fetch an approved diagnostic profile during an incident.
Read relevant cloud metrics and traces.

The access we do not hand out includes cluster-wide secret reads, direct production database writes, and unrestricted exec into pods. There are rare reasons for each. Those reasons deserve an incident ticket and a second human looking at the command.

Our Kubernetes groups now map to service ownership. The catalog team cannot casually inspect identity, and Platform cannot claim every namespace is somebody else’s concern.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: catalog-oncall
  namespace: catalog-prod
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log", "events"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets"]
    verbs: ["get", "list", "watch", "patch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: []

The empty pods/exec permission is deliberate. We had one incident in May where an engineer fixed a live configuration file inside a container, then forgot it existed. The next rollout erased it, as containers are designed to do. We spent forty minutes rediscovering the same failure with more coffee involved.

Kubernetes has plenty of sharp edges here, so we keep the policies close to the upstream RBAC model rather than inventing an internal permissions language.

“Why Can’t Platform Just Own Every Deployment?”

Platform can own the deployment path; it cannot own the operational knowledge of every service.

We tested the “Platform approves production” model last autumn. It looked safe on a process diagram. In practice, it put two platform engineers between every product team and a release, including changes that only altered invoice wording or a CSS bundle. The queue grew, people started sending urgent messages, and approval became a ritual performed after somebody had already decided to deploy.

That is gatekeeping with better YAML.

The service team now owns whether its code should go live. Platform owns the tooling that makes the decision visible, repeatable, and reversible: the GitHub Actions templates, the Argo CD projects, the cluster policies, and the audit trail. Security owns the controls that must be true before the workflow can issue production credentials. Those boundaries are written down because informal agreement lasts exactly until the person who remembers it is on holiday.

We use CODEOWNERS for files that change shared deployment behavior. A service team can change its replica count or an application environment variable. A change to a shared ingress class, an organisation-level action, or a production NetworkPolicy requires Platform review.

# .github/CODEOWNERS
/.github/workflows/reusable-deploy.yml  @platform-engineering
/platform/network-policies/            @platform-engineering @security-engineering
/services/catalog/                     @catalog-team
/services/release-api/                 @billing-team

GitHub documents the awkward parts of CODEOWNERS matching and branch protection, including the fact that a file can have one visible owner line while several teams assume they own it. We found two of those in the first review.

The rule is not “Platform stays out of releases.” We still join risky database changes, regional failovers, and anything touching shared ingress. We simply refuse to become the human merge queue for routine service work.

“What Does ‘You Build It, You Run It’ Mean On Pager?”

It means the team that changes a service takes first call for its user-facing failures, with Platform covering failures in the shared runtime.

We had to make that sentence more precise than it sounds. “Run it” had become a polite way to tell application engineers that they owned every alert, including node pressure, expired cluster certificates, and a broken Fluent Bit daemonset. Nobody learns much from a pager at 03:00 that says “the whole platform is vaguely unhappy.”

Our current split is based on the failing layer. If catalog-api returns 5xx responses because of a bad release, Catalog gets paged. If every namespace loses DNS resolution, Platform gets paged. If the application cannot reach Redis because its own connection pool is exhausted, Catalog gets paged; if the Redis service itself is unavailable, the data platform rotation gets paged and Catalog gets an incident notification.

That sounds obvious after writing it down. It was not obvious in PagerDuty, where 19 services had a generic escalation policy called production-critical.

We changed the alert annotation format so it carries an owner, a runbook URL, and a statement of what user action is failing. The last field has reduced some thoroughly unhelpful alerts.

annotations:
  owner_team: catalog
  runbook: https://ops.example.internal/runbooks/catalog-checkout-errors
  user_impact: "Customers cannot add an item to an existing order"
  dashboard: https://grafana.example.internal/d/catalog-api

Mina, who has been on our platform rota longer than the current Grafana folder structure, objected to putting ownership in annotations. Her point was reasonable: labels rot. She was right, so the alert rule now validates the owner against our service catalog during CI. We still have stale runbooks, but now they fail a check instead of waiting quietly for a bad night.

For alert design, we keep returning to the Google SRE Workbook guidance on paging: page for urgent user impact, not every noisy internal symptom. It has saved us from several clever alerts that nobody could act on.

“Are DORA Metrics Going To Become Another Scorecard?”

They will if we publish team rankings, so we do not.

We track deployment frequency, lead time, change failure rate, and time to restore because they help us locate friction in the delivery system. The DORA research is useful as a vocabulary. It does not grant permission to declare that the team with fewer deployments is failing.

Our billing team deploys release-api three or four times a week. The reporting team deploys its batch export service about twice a month, often after waiting for a partner’s sample file. Comparing those numbers is managerial astrology.

Instead, we review metric changes against specific events. In July, median lead time for Catalog rose from 7 hours to 31 hours. The cause was not developer speed. A required image scan had begun waiting behind the same two self-hosted runners that handled integration tests. We added two runners, separated scan jobs from test jobs, and lead time dropped the following week.

Change failure rate exposed a less comfortable issue in Billing. Their rate was 18% over six weeks, mostly because a database migration and application release were approved separately. The failed release-api deployment belongs in that number. We are not using it to tell the team to make fewer changes; we are using it to fund the work of joining the changes safely.

We show rolling twelve-week trends and attach incident links to failed changes. Nobody gets a traffic-light score. If a metric appears in a performance review, people will optimise the number and hide the problem. We have enough experience with that particular species of spreadsheet.

“Should We Put Argo Rollouts In Front Of Every Service?”

We don’t know yet, honestly.

We have Argo Rollouts running for three HTTP services: catalog-api, release-api, and notifications-webhook. The canary step sends 10% of traffic to the new ReplicaSet, waits five minutes, then checks request error rate and p95 latency. If either crosses the configured threshold, it aborts.

strategy:
  canary:
    steps:
      - setWeight: 10
      - pause:
          duration: 5m
      - analysis:
          templates:
            - templateName: checkout-http-health
      - setWeight: 50
      - pause:
          duration: 10m
      - setWeight: 100

The release-api incident would probably have failed its canary analysis. “Probably” matters. Our staging traffic is too thin to produce a meaningful request-error signal, and production canaries only help when the issue appears in the first slice of traffic. A missing network permission would have done so. A quarterly invoice batch failure would not.

We have not tried it past 40 nodes, and our production clusters have 27, 31, and 38 worker nodes. The controller overhead looks modest in our tests. That is not evidence for what happens after the acquisition team’s services arrive, assuming that deal survives the budget meeting.

We also do not want rollout machinery around every workload. A nightly reconciliation Job does not need traffic shifting. Neither does a CronJob that fetches exchange rates at 04:15. For these, a failed run needs a clear alert and an easy rollback of the image tag. Adding canary objects would make the repository busier without reducing the relevant risk.

Argo’s analysis templates are useful because they force us to state what “healthy” means. The hard part is choosing a metric that reflects users rather than deployment optimism.

“How Will We Know The New Release Rules Are Working?”

We’ll know when a service team can explain a failed production release from one pull request, one deployment record, and one incident timeline.

That is our test for the next six weeks. Each production deployment must link its commit, container digest, migration version, feature flags, smoke-test result, and rollback action. We are adding those fields to the release event emitted into OpenTelemetry, then building the incident view from that event instead of asking people to paste screenshots into Slack.

The first version will be ugly. It only held together this week because the billing team already had consistent Git tags and the migration workflow happened to emit a version number. Catalog does not yet do either. We are fixing the release metadata there before we announce a company-wide standard and discover that standards are easiest when somebody else has done the plumbing.

We do not need a 64th deployment message.

The remaining argument is whether a NetworkPolicy change should block a service release until its owning team approves it, or whether the platform team should approve it as shared infrastructure. We have put that decision on Thursday’s architecture agenda, where it will receive the traditional amount of adult supervision.

Kanban After 12 Months Of Interrupt-Driven Work

DevOps Oasis — Sat, 22 Aug 2026 05:41:07 +0000

One year after replacing sprints, our platform team kept the limits and lost the theatre.

On 19 August 2025, we deleted the remaining two-week sprint from our platform board while seven tickets were still marked “in progress.” Two were waiting on product teams, one was a certificate renewal nobody had planned, and one was a fairly ordinary Terraform change that had somehow acquired four assignees.

We had spent six months pretending this was a planning problem. It was an interruption problem.

Our nine-person platform team supports 36 production services, a shared Kubernetes cluster, identity, CI runners, and the collection of internal tools that only become visible when they fail at 03:00. Sprints made the work look orderly for roughly a day and a half. Then an incident, an access request, or a release deadline arrived and the board became historical fiction.

We moved to kanban with a hard WIP limit, an expedite lane, and a promise to stop calling unfinished work “carryover” as if it had merely missed a train. One year later, we’d grade the decision a B. The board changed how we start work and how we argue. It also created chores we had not put on anybody’s capacity plan.

We Expected The Board To Expose Hidden Work

The expected gain was embarrassingly simple: show the work we were already doing.

Before kanban, an engineer could have a sprint ticket, two Slack requests, a deployment rollback, and a “quick look” at a customer’s IAM policy. Only the sprint ticket appeared in the report sent to leadership on Friday. We were measuring planned work against a week that contained a great deal of unplanned work, then acting surprised when the ratio looked bad.

We expected WIP limits to make people finish things before picking up more things. Our original rule was eight active standard tickets across the team, with one separate expedite slot. Eight felt generous. It was based on the number of engineers minus the people we assumed would be in meetings, which is not a calculation we’d recommend preserving in a museum.

We also expected the classes of service to stop being decorative labels. “Expedite” meant production impact or a security deadline with fewer than five days left. “Fixed date” meant a date imposed by something outside our team: a contract renewal, audit evidence, a vendor certificate expiry. Everything else entered the standard queue.

The Kanban Guide is admirably plain about starting with the current way of working. We took that seriously, perhaps because we did not have enough spare energy to redesign the department and install a new board in the same quarter.

Our old sprint plan behaved like a railway timetable after a signal failure: technically still present, increasingly irrelevant.

The first version had three columns: Ready, Doing, Done. We added Blocked on the second day because “Doing” was hiding work that no one could move. That should have told us something about the first version’s intellectual depth.

Tuesday Replenishment Changed More Than Stand-Up

What actually happened was quieter than we expected. Nobody stood up in a meeting and declared a new era of flow. The useful shift came on the first Tuesday when the Ready column held 19 tickets and Doing had reached its limit.

We had 11 active tickets. More precisely, we had 11 tickets marked active; three were waiting for a reply, two had not been touched since the previous week, and one was a task to “investigate service mesh options,” which is a sentence that should trigger an automatic timeout.

The team started asking a different question: what can we finish before taking this? That sounds obvious when written down. In practice, it changed the order of conversations. An engineer working on an OpenTofu module asked the service owner for the missing variable list before beginning a new request. Another closed a stale upgrade card after confirming the vendor had withdrawn the affected release. We stopped treating context switching as evidence of responsiveness.

Our daily check-in dropped from 25 minutes to around 11. We walk the board right to left, starting with blocked items and then work closest to done. We do not go person by person. The person-by-person version had become a polite recital of activity, and everyone knew it.

Throughput settled around 14 completed standard tickets per fortnight, though the number is lumpy enough that we do not use it for individual targets. In October, we completed 23 because several old requests were finally closed. In January, we completed six while rebuilding the GitHub Actions runner group after an image change broke our ARM jobs.

Little’s Law gave us a useful way to discuss that variation: more work in progress generally means longer time to finish, assuming the system is reasonably stable. The formula is short; the hard part is refusing to add “just one more” item when the board is full. The explanation of Little’s Law is more useful than most delivery dashboards we’ve purchased.

This only held because our engineering managers agreed not to insert work directly into Doing. They can request an expedite item. They cannot assign it a person and declare the WIP limit an inconvenience.

The Blocked Column Made Dependencies Public

The surprise nobody predicted was that blocked work would become mildly political.

A ticket marked Blocked tells a different story from a ticket left in Doing. It tells us the platform team is waiting on a security approval, a product decision, or a vendor reply. That visibility is useful, but it also names the place where work has stopped. Some teams welcomed it. One product group asked whether we could use a softer status because “blocked sounds accusatory.”

We did not.

We did change the policy. Every blocked ticket now needs a short reason, the name of the waiting party, and a next follow-up date. “Waiting on product” is not enough. “Waiting on Checkout to choose whether webhook retries should be at-least-once or deduplicated; follow up Thursday” is enough.

By November, the board showed an uncomfortable pattern. Access work was not slow because creating roles in Okta took long. It was slow because requesters submitted a ticket before deciding which people needed access and why. We had built a form that asked for an application name and environment, then left the actual approval chain to comments.

That led to a small but worthwhile change: access requests now require an owner and an expiry date before entering Ready. The number of access tickets did not fall. Their median elapsed time fell from 9.4 business days to 3.1, mostly because we stopped accepting half-formed requests.

The blocked data also gave us evidence during the February incident review. We could show that six “urgent” observability requests had waited on service owners to define an SLO. The Google SRE workbook’s alerting guidance says alerts should connect to a user-facing objective. We had been asking teams for exactly that, then allowing the cards to enter the queue without it.

There is still a social cost. People dislike seeing their team named in a blocked reason, even when the wording is factual. We have not found a better answer than being precise and not turning the board into a blame register.

The Board Needed Someone To Weed It

Nobody budgeted for board maintenance.

Not administration in the abstract. Actual maintenance: closing dead cards, correcting issue types, checking automations, chasing owners, and making sure a ticket moved to Done reflects a deployed change rather than a pull request waiting for approval.

For the first three months, GitHub Projects automation handled enough of this that we thought we had escaped the usual board janitor role. A pull request merge moved the linked issue to Done. New issues with the platform label entered Triage. An incident label put work into Expedite.

Then reality arrived. A merged pull request could deploy to staging and sit there for nine days. Incident tickets kept the label after the incident was closed. A request from Finance inherited platform because someone copied an old issue template. The board was accurate in the same way a kitchen drawer full of old cables is organised: there was a system, once.

Here is how the setup drifted over the year:

# 19 August 2025
workflow:
  columns: [Ready, Doing, Done]
  doing_wip_limit: 8
  expedite_wip_limit: 1
  done_when: "pull request merged"
automation:
  label_platform: "add to Ready"
  label_incident: "move to Doing"

# 19 August 2026
workflow:
  columns: [Triage, Ready, Doing, Blocked, Verify, Done]
  doing_wip_limit: 6
  expedite_wip_limit: 1
  blocked_requires: [reason, waiting_on, follow_up_date]
  done_when: "production verification recorded"
policies:
  triage_owner_rotation: weekly
  stale_ready_after_days: 30
  expedite_requires: "active customer impact or security deadline under 5 days"
automation:
  label_platform: "add to Triage"
  label_incident: "move to Expedite"
  merged_pull_request: "move to Verify"
  verified_production: "move to Done"

We now rotate a weekly triage owner. It takes about 35 minutes on a normal week and longer after an incident. The owner checks new work, pings stalled cards, and closes tickets that have become archaeology. Nobody loves the rota, but it is far less annoying than discovering during replenishment that Ready contains a request for Ubuntu 20.04 package support from last winter.

Our on-call rota deserves a brief mention here. We stopped making the on-call engineer triage owner after one week in September, when the same person had a noisy Kafka page at 02:17 and a backlog of 14 intake tickets by breakfast. The board can wait. A tired engineer should not have to decide whether a request is “standard” or “fixed date” before coffee.

We Kept Limits And Replenishment, Not Velocity Charts

The pieces we kept are the ones that force a decision.

The Doing limit is now six, not eight. We lowered it after measuring elapsed time and noticing that work got stuck at six anyway; the extra two slots were simply permission for two more partially started tasks. Six feels restrictive on Monday morning. By Thursday, it usually feels sensible.

We also kept twice-weekly replenishment. Tuesday and Friday, 20 minutes each, with the triage owner and whichever engineers are free. We pull enough Ready work to keep the board supplied, not enough to make everyone feel booked. We do not estimate every card. If a task is clearly larger than about five working days, we split it or write a short discovery card with a specific decision as its exit condition.

Our preference is blunt: we dislike story points for platform work. Giving an S3 lifecycle policy a five and a Kubernetes minor upgrade a thirteen produced elaborate certainty theatre. Elapsed time, blocked age, and a short list of current work tell us more.

We retained a monthly service-delivery review, although we cut the slide deck from 18 slides to four screenshots and a list of the oldest blocked items. We look at completion times by work type, not individual engineer. The point is to see that fixed-date work is clogging the queue or that access requests are incomplete, not to invent a ranking system for people who happened to draw the ugly tickets.

We have not tried this past 40 nodes or across multiple platform teams. Our cluster has 17 worker nodes and one team owns the board. At a larger scale, shared services and dependency boards may require a different arrangement. We are not claiming a magic ceiling; we simply do not have evidence beyond our own rather manageable mess.

We Ripped Out Swimlanes And Deadline Colours

We removed the swimlanes.

At one point the board had lanes for Kubernetes, CI, cloud accounts, observability, security, and “team enablement.” Cards duplicated across them, work vanished below the fold, and nobody could tell whether the WIP limit applied to the lane or the column. A board that needs a legend has already lost an argument with its users.

We also removed due-date colours. Red cards encouraged people to work on the nearest date even when the date was invented by a requester who wanted attention. Fixed-date work now has a documented external deadline and a reason. A request for “this sprint if possible” is standard work, even if the ticket title contains three exclamation marks.

The final thing we ripped out was the rule that every engineer must always have a card assigned. Pairing on a thorny rollout, reviewing a production verification, or helping the on-call engineer is work. It does not become more real because a board has somebody’s avatar attached.

One argument remains unresolved. Security remediation gets its own expedite slot when there is an active exploit, but we still disagree about high-severity findings with no known exploit. Half the team wants a second reserved slot. The rest of us think that would quietly become a permanent bypass around the limit. The next critical CVE will probably settle it for us.

Leadership Mistakes We Put Into Production Before Lunch

DevOps Oasis — Sat, 22 Aug 2026 05:40:47 +0000

Six management failures we shipped alongside perfectly valid infrastructure changes.

At 11:42 on Monday, the billing-export pull request had four approvals, one frustrated staff engineer, and no merge. Nothing in the diff was controversial. It changed a retry timeout from 30 seconds to 45. The problem was that our repository rules required a platform lead to approve anything touching infra/, and both platform leads were in a quarterly planning meeting.

We wrote that rule six months earlier. We then spent three hours discovering what it had actually done to the team.

We Made Leads The Default Merge Queue

What we did

We put @platform-leads in CODEOWNERS for most infrastructure paths. Any Terraform module, Kubernetes manifest, Helm chart, or GitHub Actions workflow waited for one of two people.

# Bad: .github/CODEOWNERS
/infra/              @platform-leads
/kubernetes/         @platform-leads
/.github/workflows/  @platform-leads

It looked orderly in the repository settings screen. We had senior review on high-impact changes. Nobody could accidentally modify production wiring on a quiet Friday afternoon. The fact that “quiet Friday afternoon” is when people make choices they regret did occur to us.

Why it looked reasonable

We had just hired three engineers who had not worked with our AWS accounts before. At the same time, a state-lock incident had left an S3 bucket policy too open for 47 minutes. We wanted a brake.

GitHub’s CODEOWNERS rules make this easy to set up, and easy to overuse. A broad path ownership rule feels like governance because it produces approvals. It doesn’t tell us whether the approval added useful scrutiny.

What it cost us

Between 4 and 15 August, 31 pull requests waited longer than two business hours for a required reviewer. Nine were harmless dependency bumps. Six were fixes to alerts already paging people. One was the retry timeout.

The less obvious cost was learned helplessness. Engineers started asking, “Can you just take this?” before they had read the plan output. Leads got worse at reviewing because our queue had become full of changes too small to deserve deep attention.

Our review time became an interrupt-driven service, which is a fairly miserable thing to run.

The fix

We moved ownership to the people who operate each service and kept lead review for a short list: identity, network boundaries, account creation, and production deletion paths.

# Corrected: .github/CODEOWNERS
/services/billing-export/  @billing-export-owners
/modules/retry-policy/     @platform-runtime
/modules/iam-boundary/     @platform-leads
/infra/accounts/           @platform-leads

We also added a risk:high label that requires one lead approval, rather than pretending every YAML file carries the same consequence. We pinned terraform 1.8.5 in .tool-versions on 2026-08-18, so reviewers were at least looking at plans produced by the same binary. Tool drift had been supplying enough fake disagreement already.

The new rule only held because our service owners already had production access and knew their own rollback paths. We have not tried it beyond 40 engineers or across a company with three separate compliance teams. We’d expect the ownership map to become its own small administrative animal there.

We Praised Firefighting In The Incident Channel

What we did

During the 03:18 orders-api incident on 7 August, a senior engineer found that an Envoy sidecar image had been pulled with a stale digest. They manually restarted six pods, traffic recovered, and the incident channel filled with grateful emoji.

Then we praised the restart in the weekly all-hands.

We did not praise the engineer for writing down the image-cache condition, because that happened two days later in a quiet document that fewer people read. We effectively taught the group that visible heroics counted more than removing the reason for heroics.

Why it looked reasonable

Incidents are tense, and people deserve credit when they act calmly under pressure. The restart did restore checkout. Nobody wants leaders to respond to a hard night with a lecture about process.

We also had a habit from smaller teams: the person who fixed the immediate problem was usually the person who could fix the whole problem. That stops being true when the system has 26 services, four cluster node pools, and a deployment controller that behaves differently at 03:00 than it did in staging. Or at least it feels that way after coffee.

What it cost us

Within ten days, the same stale-digest condition appeared in catalog-sync. The engineer on call restarted pods again because it was the known move. We lost 19 minutes of delayed catalogue updates and spent 14 person-hours on two nearly identical incident calls.

Worse, two people who had noticed the cache behaviour before the first incident did not bring it up in the review. They assumed an emergency workaround was the approved answer.

The fix

We changed our incident follow-up language. The incident commander now names three separate contributions: containment, diagnosis, and prevention. They are often done by different people.

Our postmortem template also includes “what made the fast workaround attractive?” That wording matters. “Why did we not fix it properly?” produces defensive writing and suspiciously clean timelines. The Google SRE postmortem guidance has the right instinct here: get the facts down without making the document a trial.

For the image issue, we set imagePullPolicy: Always for the affected deployment class, added digest validation to the release job, and created a runbook that says a restart is containment only. It is deliberately blunt.

If your team celebrates the person awake at 03:18, celebrate the person who makes 03:18 boring next month too.

We Assigned Pager Duty By Reporting Line

What we did

Our escalation policy routed alerts from payments, fulfilment, and customer-profile through the platform manager after the primary responder. The manager had enough context to coordinate work, so the routing felt sensible.

It also meant a people manager was the automated second-line responder for systems they did not deploy or change.

# Bad: escalation policy in Terraform
rule {
  escalation_delay_in_minutes = 10
  target {
    type = "user_reference"
    id   = pagerduty_user.platform_manager.id
  }
}

Why it looked reasonable

The platform manager attended architecture reviews, knew which teams were stretched, and could wake the right person. On paper, that resembles escalation.

We confused being informed with being useful at 02:00. Those are different jobs. One of them needs a laptop, current access, and enough familiarity with payment-webhook to distinguish a bad certificate from a bad deploy.

What it cost us

On 12 August at 01:56, a latency alert reached the manager after an on-call engineer was already investigating. The manager had to search the rota, message the service owner, and then relay a dashboard link. The service owner joined 17 minutes later.

No individual made a bad call. The policy encoded a slow handoff and asked tired people to compensate for it.

It also damaged trust in the rota. Service owners felt paged by proxy while platform absorbed blame for alerts outside its control.

The fix

We route to a service secondary first, then to an incident commander who can coordinate across teams. Managers get an informational notification after 20 minutes, not a wake-up call by default.

# Corrected: escalation policy in Terraform
rule {
  escalation_delay_in_minutes = 10
  target {
    type = "schedule_reference"
    id   = pagerduty_schedule.payments_secondary.id
  }
}

rule {
  escalation_delay_in_minutes = 20
  target {
    type = "schedule_reference"
    id   = pagerduty_schedule.incident_commander.id
  }
}

The PagerDuty escalation policy documentation is clear on how the mechanics work. The harder part is deciding who can act, rather than who sits highest on an org chart.

We still argue about whether managers should ever be in a daytime escalation chain. For customer-data exposure, probably yes. For a failed Kubernetes CronJob, absolutely not.

We Treated A Green Dashboard As Good Management

What we did

For most of July, our leadership update reported cluster availability, deployment completion, and ticket closure. All three were green. Meanwhile, engineers were bypassing the shared delivery path because preview environments took 38 minutes to appear.

Nobody hid that number. We simply had not made it a number we discussed with any seriousness.

Why it looked reasonable

Cluster availability was available in Grafana. Deployment completion came out of GitHub Actions. Ticket closure was in Jira, waiting politely for somebody to make a chart. These numbers were consistent and easy to report.

The preview-environment delay came from a mixture of runner queue time, Terraform state contention, and a slow image scan. It required actual investigation. We chose the numbers that arrived pre-chewed.

What it cost us

Two teams began keeping long-lived feature branches because they could not get useful feedback in a working day. One team deployed a feature flag directly to production to avoid another preview wait. That flag stayed enabled for five days because nobody was certain which branch had created it.

We had technically healthy infrastructure supporting increasingly unhealthy delivery habits.

The fix

We replaced the status slide with a weekly operating review of four measures: time from merge to production, failed deployment recovery time, on-call pages per service, and age of the oldest unowned alert. We use the definitions from DORA’s research program as a starting point, then adjust when the measurement lies.

For example, “merge to production” excludes changes waiting behind a scheduled customer release. Counting that delay as engineering delivery time would make a calendar meeting look like a platform defect.

We removed ticket closure entirely. We dislike it as a leadership metric. It rewards chopping work into small rectangles and then admiring the pile.

We Called On-Call A Growth Opportunity

What we did

When a product engineer joined the rota for the first time, we described it as a chance to learn the system. It was a chance to learn the system. It was also a week of interrupted sleep, a 06:40 database connection alert, and a Friday afternoon spent catching up on work they had planned to finish Tuesday.

We had no written rule for time back after a bad night.

Why it looked reasonable

Shared on-call builds ownership. We wanted product teams to see the consequences of their release decisions instead of handing every production concern to platform. That preference remains.

The mistake was treating the learning benefit as payment for the disruption. People do not need a motivational slogan after being woken by an alert caused by a threshold someone else set in 2023.

What it cost us

Three engineers swapped out of the rota in the next quarter. One did so quietly, citing “capacity.” We read that as a scheduling issue until their manager told us they had been paged five times in two weeks and had received no protected recovery time.

Alert quality also got worse because responders deferred cleanup work. Nobody volunteers to improve an alert that might later wake them if the work competes with a sprint commitment.

The fix

A page between 23:00 and 06:00 now earns a late start or half-day recovery without negotiation. A second page in the same night triggers a manager check-in and a review of the alert within five working days.

We also publish the rota alongside expected service work, so a person carrying orders-api is not assigned a migration cutover that week. This costs delivery capacity. Good. The capacity was fictional before; it was being borrowed from somebody’s sleep.

We Replaced Decisions With Status Meetings

What we did

Our Thursday platform meeting had 14 people, 45 minutes, and a standing agenda full of updates. The recurring item was “progress on secrets migration.” For six weeks, everyone reported progress while nobody decided whether service teams had to move by the September deadline.

The migration repository had 63 open issues. Twelve were blocked on choices that no ticket owner could make.

Why it looked reasonable

Status meetings make managers feel informed. They also give people a place to raise concerns, which sounds generous until the same concern appears in six consecutive agendas with no owner assigned to resolve it.

We had been trying to avoid unilateral decisions after a previous migration upset two application teams. Consensus became the safer social move. It was also much slower.

What it cost us

By the time we set a policy, the legacy Vault token format had spread to two new services. The team then had to support both formats during the busiest release month of the quarter.

The meeting itself cost 31 person-hours over those six weeks. The delay cost more, but the meeting cost was easier to calculate and therefore more annoying.

The fix

Every recurring operational meeting now has a decision log with three fields: decision required, named decider, and deadline. Updates go in writing before the meeting. If no decision is pending, people get 45 minutes back.

For the secrets migration, we chose a hard date: new services after 2026-09-15 must use workload identity; existing services get a supported adapter until 2026-12-31. We accepted that two teams would dislike the date more than they disliked ambiguity.

The adapter is still ugly. We have not found a clean migration path for the vendor service that only accepts a static credential, and nobody is excited about the options.

SRE Starts With The Alert That Woke Us

DevOps Oasis — Sat, 22 Aug 2026 05:40:26 +0000

How we made service reliability a team job, not an on-call punishment.

The payment-api alert was still firing when we opened the incident channel, and nobody could tell whether customers were actually blocked or whether a dashboard had found another way to ruin breakfast.

SRE means site reliability engineering: a way of running software where the people who build services also set clear expectations for how those services should behave in real use. We measure whether they meet those expectations, respond when they do not, and spend planned time removing repeated sources of failure.

At 08:06, we had a red graph, four Slack replies, and no useful answer to the only question that mattered: could a customer complete a payment? That gap is why we started treating SRE as a working practice rather than a title somebody receives after learning enough Kubernetes nouns.

Start With What A User Can Actually Do

A service can have healthy pods, low CPU use, and a very cheerful deployment status while still failing the one task people came to perform. We learned this with payment-api in April. Its /healthz endpoint returned 200, every container was running, and the PostgreSQL connection pool looked normal. Meanwhile, requests to create a payment were waiting 18 seconds on a call to our fraud provider.

Nobody buying a train ticket cares that our health endpoint is fine.

For SRE work, we begin with a user action: create an account, load a report, upload a document, complete a payment. That action gives us something concrete to measure and protect. It also stops teams from treating infrastructure metrics as proof that an application works.

We wrote down three actions for the payments service:

A customer can submit a payment.
A merchant can retrieve its payment status.
Our nightly settlement worker can send a batch to the bank before 02:00.

Those are not equally valuable. A delayed merchant status page is annoying. A payment request that disappears or hangs is a financial incident. Settlement missing its bank window has a different kind of unpleasantness, usually involving spreadsheets and a person from finance who has every right to be annoyed.

The Google SRE book’s section on service level objectives is useful here because it separates what users experience from whatever machinery happens to sit behind it. We borrowed the idea, then made the wording much plainer in our own service docs.

A user action is the unit we protect; the rest is evidence.

Measure Requests, Not The Mood Of The Cluster

Once we had a useful user action, we needed a number. For payment submission, we count successful HTTP requests and failed HTTP requests. A request is successful if it returns a 2xx response within five seconds. A 500, a timeout, or a response after five seconds counts as bad.

That five-second line is not sacred. We picked it after looking at 30 days of request timing. Most payments completed in 430–900 ms. At 2.5 seconds, support tickets started to correlate with retries. At five seconds, our web client gave up and showed the customer an error. We chose the point where the customer’s experience had already become poor, rather than pretending 400 ms was a promise we could keep during the Monday morning rush.

In SRE terms, the thing measured is a service level indicator, usually shortened to SLI. We call it “the payment success rate” in conversation because acronyms spread quickly enough without our help.

This Prometheus query is close to the one on our dashboard:

sum(rate(http_requests_total{service="payment-api",code=~"2.."}[5m]))
/
sum(rate(http_requests_total{service="payment-api"}[5m]))

The query is deliberately boring. It counts good requests divided by all requests. It does not estimate customer happiness from node load or count pods with names containing payment. Those metrics still have a place when diagnosing a fault, but they are poor promises to make to users.

We also exclude requests made by our synthetic checker from this ratio. Otherwise a healthy checker can keep the graph looking better while ordinary traffic is failing, which is exactly the sort of technical correctness we dislike.

A useful SLI maps a visible user action to a number we can inspect during an incident.

Set A Target You Can Defend In A Meeting

A service level objective, or SLO, is the target for that measurement over a period of time. Ours for payment submission is 99.9% successful requests in a rolling 28-day window.

That means we permit roughly 43 minutes of failed or slow request time per month if every request fails at once. Real faults are messier than that: a partial outage may consume the allowance across thousands of requests in ten minutes. Still, the number gives us a shared boundary.

We tried 99.99% first.

It did not work.

The target sounded suitably serious in the planning document, but it gave payment-api about four minutes and 19 seconds of failure allowance in 30 days. A routine database failover consumed most of that. A deploy that caused a two-minute spike put us in a state where the sensible response was “stop changing anything,” including a fix for the thing that had just failed. We changed it after two months. Nobody has suggested going back.

An SLO should reflect the cost of failure and the cost of making change. For payment submission, 99.9% is defensible because a short failure is costly, but we also deploy fixes and dependency changes every week. For our internal analytics export, 99.5% is plenty. The report is useful at 09:00; it does not need a ceremonial response at 03:00 because one CSV was 12 minutes late.

The target is a negotiated engineering decision, not a number copied from a cloud provider’s brochure.

Treat The Error Budget As Permission To Change

The difference between the target and actual performance is called an error budget. If our target is 99.9%, the remaining 0.1% is the amount of bad service we have agreed can happen in the measurement window.

That phrasing can sound grim. We prefer to see it as permission to take normal engineering risks. Deployments, schema changes, cache migrations, and dependency upgrades all carry some chance of disruption. If we demand zero disruption forever, we get slower changes, stale dependencies, and people quietly avoiding work that needs doing.

At 11:35, during the payment incident, the budget had fallen from 71% remaining to 63%. That gave us room to ship a timeout change and route a small percentage of traffic around the fraud provider. Had we only had 2% left, we would have made a different call: freeze feature releases, use the safer rollback path, and put the reliability work at the top of the sprint.

We keep the policy small:

payment-api:
  slo: 99.9
  window: 28d
  budget_remaining_below: 10%
  release_action: "approval from on-call and service owner"

This is not an automatic deployment lock. We tried a hard lock in GitHub Actions last year, and engineers worked around it by classifying urgent changes as operational patches. The YAML survived; the lock did not.

An error budget gives release decisions a number to argue from instead of relying on whichever person is most worried in the call.

Page People Only When They Can Act

An alert should tell an awake person that they need to do something now. Not investigate later. Not admire a graph. Not acknowledge it so that the alert noise reduces by one.

Our old alert set had 146 rules for a cluster of 24 nodes. At 02:40 one Thursday, the on-call engineer received 17 pages for a single Redis failure: node disk pressure, pod restarts, connection errors, cache miss rate, eviction count, and several alerts that had apparently been written by a committee of anxious refrigerators.

We cut that down. The page now fires when the payment SLO is burning quickly enough that we are likely to consume 5% of the monthly budget in one hour. The disk and restart alerts still exist, but they go to a working-hours channel unless they are directly causing user failure.

This uses the “burn rate” approach described in the SRE workbook’s alerting chapter. We like it because it joins urgency to impact. A five-minute blip at 03:00 can wait. A failure eating a month’s allowance before lunch cannot.

The alert notification includes three links: the SLO graph, the relevant deployment history, and the runbook. It does not contain fourteen labels, a container hash, and the hostname of a node that may no longer exist by the time somebody reads it.

The runbook is where we put immediate checks, known failure modes, and safe rollback steps. It is not a wiki page titled “Payments Operational Knowledge” last edited in 2022 by someone who moved teams.

A page earns its place on the rota only when an engineer has a real action available.

Reliability Work Has To Compete For Sprint Time

SRE fails when reliability tasks become “we’ll get to it after the roadmap.” There is always another roadmap item. There is also always another slow query, expiring certificate, or queue consumer that only fails after a long weekend.

We reserve capacity in each service team’s sprint for reliability work. Not a fixed 20% in every case; fixed percentages create odd incentives and fake tickets. Instead, the service owner brings the largest repeat incident, highest-risk dependency, or most worrying budget trend to planning. We ask what would make that problem less likely next month.

For payments, the answer this quarter was not another dashboard. We had plenty of dashboards. We added a circuit breaker around the fraud provider, recorded dependency timing separately, and changed the retry policy so that 400 failed requests do not all retry in the same second.

Our colleague Mina still thinks circuit breakers are a polite name for “giving up early.” She is partly right. We are still arguing about whether failed fraud checks should enter a manual review queue during a provider outage or be rejected immediately. That decision crosses risk, support staffing, and customer trust; no latency graph will settle it for us.

We track reliability work in the same backlog as product work. If it needs a separate spreadsheet to exist, it is already losing.

Words We Use Before They Become Meeting Noise

A short glossary helps because SRE has enough abbreviations to make ordinary incident calls sound like a radio test.

Service level indicator (SLI): The measurement of a user-facing action. For us, payment requests completed successfully within five seconds.

Service level objective (SLO): The target for that measurement over a time window. Our payment submission target is 99.9% over 28 days.

Error budget: The allowed amount of failure implied by the SLO. It is what remains when actual performance is compared with the target.

Burn rate: How quickly the service is spending its error budget. A fast burn rate deserves attention even if the monthly number still looks acceptable.

On-call: The person scheduled to respond when an alert pages. This should rotate, include proper handover, and not quietly become one person’s permanent second job.

Runbook: A practical page for responding to a known alert: where to look, what is safe to change, and when to escalate.

Prometheus has a clear alerting rules reference if you are building alerts from scratch, while Alertmanager’s grouping documentation explains how to avoid receiving seventeen versions of the same bad news.

Words are useful only if the person holding the pager can turn them into a decision.

Keep The First Version Small Enough To Maintain

We have not tried this approach past 40 nodes or across dozens of independently deployed teams. At our size, one platform group and six service teams can still agree on common labels, a dashboard layout, and who owns an alert. Larger organisations will need more structure, and probably more patience than we had at 08:06.

Start with one service that matters, one user action, one measured target, and one page that somebody can act on. Do not wait for a grand reliability programme. We began with payment submission because its failures were visible, expensive, and already waking people up.

By 16:40, the fraud provider had recovered, the error budget had stopped falling, and the incident channel had gone quiet. We still have a node_filesystem_avail_bytes alert that wakes people too often on a particular worker pool. It has been open for 19 days, which is annoyingly longer than the incident that made us write this post.

Ansible Myths We Met During The Debian 12 Cutover

DevOps Oasis — Sat, 22 Aug 2026 05:40:06 +0000

What held up across 28 hosts, and what absolutely did not.

02:13.

That was when the last Debian 11 VM in our payments group stopped accepting the old OpenSSH cipher list, halfway through an Ansible run that had looked boring for 41 minutes. Boring is usually a compliment in infrastructure work. At 02:13, it became a theory.

We moved 28 application and utility hosts to Debian 12 over three evenings: 14 API nodes, six Celery workers, four reporting boxes, two Redis replicas, and two machines nobody could name without checking NetBox. We ran ansible-core 2.17.7 from a pinned Python 3.12 virtual environment on our control host. The exercise produced a familiar set of opinions from people who have used Ansible once, inherited it reluctantly, or seen a YAML file commit a minor crime.

Most of those opinions contain a grain of truth. Grains are how people end up eating sand.

“Ansible Is Just SSH With YAML”

Verdict: false, though SSH is still where many bad evenings begin.

For ordinary Linux hosts, Ansible’s default transport is SSH, and that fact encourages people to dismiss the rest as a collection of remote shell commands with more indentation. We did use SSH. We also used inventory groups, facts, idempotent modules, handlers, privilege escalation rules, check mode, variable precedence, and a run history we could inspect after someone asked why cron had restarted on a reporting node at 01:47.

Calling all of that “just SSH” is like calling PostgreSQL “just files on disk.” Technically defensible, professionally unhelpful.

The distinction mattered during the cutover because our hosts were not alike. The API nodes run gunicorn behind Nginx. Celery workers have different systemd limits. The reporting boxes mount an NFS share that the application hosts never see. A shell loop would have needed its own logic for each difference, along with error handling and a way to tell whether a retry had changed anything. We have written those loops before. They age like yoghurt in a server room.

Here is the small part of the play that put the right APT source on each Debian 12 host and notified Nginx only when its file changed:

- name: Configure internal package source
  ansible.builtin.deb822_repository:
    name: oasis-internal
    types: [deb]
    uris: "https://apt.devopsoasis.internal/debian"
    suites: ["{{ ansible_distribution_release }}"]
    components: [main]
    signed_by: "https://apt.devopsoasis.internal/keys/archive.gpg"
    state: present
  notify: Reload nginx

- name: Install application packages
  ansible.builtin.apt:
    name:
      - oasis-api={{ api_package_version }}
      - nginx
    state: present
    update_cache: true

handlers:
  - name: Reload nginx
    ansible.builtin.service:
      name: nginx
      state: reloaded

The Debian repository module and the APT module gave us useful state instead of an optimistic curl | tee sequence. We could rerun the play after fixing the cipher issue and see 23 hosts report ok, four make the expected package changes, and one fail on a stale internal repository key.

That said, Ansible does not remove SSH’s sharp edges. A stale known_hosts entry, a host key rotation, an expired jump-host certificate, or a control socket left under /tmp can still stop a run before a task begins. We had all four categories represented in the preceding month, because apparently our infrastructure enjoys variety.

Ansible gives SSH a memory and a grammar. It does not grant SSH better manners.

“If A Playbook Is Idempotent, Rerunning It Is Safe”

Verdict: dangerously false.

Idempotence means a task should reach the same desired state when applied repeatedly. It does not mean every action surrounding that task is harmless. A package install can be idempotent while the package’s post-install script restarts a service. A template can be idempotent while its handler drains a connection pool at a bad moment. A database migration can be “already applied” and still leave a node with an incompatible binary if deployment order goes wrong.

We had a clean example on the Celery workers. The role installed oasis-worker, rendered a systemd unit, and restarted the service when either changed. On Debian 11, the worker used a unit file with Restart=always. On Debian 12, we changed it to Restart=on-failure after discovering that a bad RabbitMQ credential caused a very energetic restart loop and 9 GB of logs over an on-call shift.

The first run changed the unit file. Good. The second run should have reported no changes. Instead, it restarted all six workers because a task used state: restarted as a convenient substitute for thinking.

We replaced it with this:

- name: Install worker systemd unit
  ansible.builtin.template:
    src: oasis-worker.service.j2
    dest: /etc/systemd/system/oasis-worker.service
    owner: root
    group: root
    mode: "0644"
  notify:
    - Reload systemd
    - Restart oasis worker

- name: Ensure worker is enabled and running
  ansible.builtin.systemd_service:
    name: oasis-worker
    enabled: true
    state: started

handlers:
  - name: Reload systemd
    ansible.builtin.systemd_service:
      daemon_reload: true

  - name: Restart oasis worker
    ansible.builtin.systemd_service:
      name: oasis-worker
      state: restarted

The handler behaviour documented by Ansible is useful here: notifications coalesce, so six changed tasks still cause one restart at the end of a play. That only helps if we notify on real changes and avoid casually restarting a process because the playbook happened to visit it.

There is a second trap: changed_when: false. People add it to quiet noisy shell tasks, then forget that they have hidden the only signal a downstream handler had. We found one role doing exactly that around update-ca-certificates. It was marked unchanged even when it had imported a new internal CA. The application unit therefore kept running with an old trust store until its next scheduled restart.

Silence in Ansible output is not proof that nothing happened.

We now treat reruns differently by class of work. Package and file convergence can run repeatedly. Service reloads need a reason. Schema changes get a separate deployment step and a named human watching the output. We still have an argument going about whether certificate rotation belongs in the same play as application rollout. Marta, who carries the Thursday primary rota, wants it split permanently. We have not settled it, mostly because nobody enjoys owning the extra release button.

“Roles Make Every Ansible Repository Easier To Maintain”

Verdict: false after the third layer of includes.

Roles are useful. We keep roles for things with a real lifecycle: oasis_nginx, oasis_api, oasis_worker, node_exporter, and vector_agent. Each has defaults, templates, handlers, and tests. That structure lets a new engineer find the file that owns /etc/nginx/sites-enabled/oasis-api.conf without hiring a tracking dog.

Then roles become a way to hide choices.

Our oldest repository had a common role that installed base packages, set SSH settings, configured journald, added users, installed monitoring, patched APT sources, mounted volumes, and once, for reasons lost to history, configured mailutils. Every host received it. Its defaults file had 67 variables. The role’s README said “foundation configuration,” which meant nobody knew whether changing it would restart half the fleet.

We removed common during the Debian 12 work. Not renamed it. Removed it.

The replacement is less elegant to people who enjoy directory trees. The base play now names what it does:

roles:
  - role: base_packages
  - role: ssh_hardening
  - role: journald
  - role: node_exporter
  - role: vector_agent

That is more repetitive, and we prefer it. A deployment plan should make its dependencies visible. When ssh_hardening changes, we can run it against the staging bastion first rather than wonder what else common believes it owns this week.

We also stopped putting application-specific variables in group_vars/all.yml. That file had become the cupboard under the stairs: Redis memory limits beside deploy keys, Nginx rate settings beside a legacy hostname, each one apparently too small to deserve a proper home. Variable precedence in Ansible is powerful enough to support almost any arrangement, which is a polite way of saying it will let us build a difficult mess. The precedence rules are clear; the harder part is declining clever arrangements before they spread.

Roles help when they represent a bounded unit of ownership. A role that changes SSH, packages, users, metrics, and mail is a landfill with a defaults/main.yml.

“Check Mode Tells Us What Production Will Do”

Verdict: partly true, and we still run it on every merge.

ansible-playbook --check --diff catches a surprising amount. It spots malformed templates, missing variables, unexpected file diffs, and package changes that would otherwise appear during a production window. For the Debian 12 migration, check mode caught an Nginx template referring to ssl_protocols TLSv1.2 TLSv1.3; on a host group where TLS terminated at the load balancer. The task was valid. The configuration was unnecessary. That distinction saved us a small amount of clutter, which is the usual reward for doing review properly.

It cannot predict everything.

Modules vary in their check-mode support. Commands and scripts cannot know their impact unless we tell Ansible with creates, removes, or explicit changed_when logic. External systems are worse. Our deployment task calls an internal release API to register the package version. In check mode, we skip it. Running a fake registration against production would turn a dry run into an incident rehearsal.

Package managers also have opinions. APT can calculate planned installs, but the final answer depends on repository contents, dependency resolution, held packages, and whether the mirror has finished syncing. On the second cutover night, our internal mirror had libpq5 15.8 for eleven minutes before its metadata caught up. Check mode at 22:06 looked fine; the real run at 22:17 installed nothing until we forced an index refresh.

This only held because our package mirror is local and the sync lag is usually under 15 minutes. We have not tried this process across 40 nodes or across regions with separate mirrors. The current serial batch size of two works because someone can read the output before the next pair begins, not because two is a mystical number.

Check mode is a review aid. Treating it as a production simulation invites a very confident failure.

“Ansible Is Too Slow For Real Deployments”

Verdict: mostly true, if “real deployments” means every host deserves parallel attention.

Ansible is not slow in the way people mean when they complain about it. A play spends time connecting, gathering facts, copying files, waiting for package locks, and asking remote services whether they already match the requested state. That work is visible. A bash fan-out script hides the same waiting behind interleaved output and gives it a name like deploy-final-v4.sh.

For 28 hosts, our full Debian 12 convergence play took 18 minutes and 36 seconds with forks = 10. Fact gathering alone consumed just under four minutes on the first run. We could turn it off, and we did for the application-only deployment play. We did not turn it off for the OS migration because several templates depend on distribution release and interface facts, and discovering that late is tedious.

The larger problem is that Ansible’s default execution model makes safe rolling work feel deliberate. We set serial: 2 on API nodes, wait for each host to return to the load balancer, and only then move on. That is slower than replacing all 14 at once. We prefer slower. The incident report from 2023, when a bad Nginx include removed health-check responses across an entire pool, remains available to anyone tempted by speed.

We do use the free strategy for low-risk work such as updating node exporter and Vector. It prevents a slow reporting host from making every other machine wait at the same task boundary. We do not use it for application releases, where matching order across a small batch is easier to observe and roll back.

Ansible has a ceiling. We would not choose it to coordinate thousands of ephemeral containers, and we dislike pretending a tool should solve jobs it was not built for. For a few dozen long-lived servers with awkward local differences, the minutes it spends checking state are cheaper than the minutes we spend explaining a broad outage.

Jenkins Versus GitHub Actions After Our 43-Minute Queue

DevOps Oasis — Sat, 22 Aug 2026 05:39:45 +0000

What we kept, what we moved, and why Jenkins still runs one awkward job.

At 10:17 on Tuesday, our release branch had 19 builds waiting behind a Jenkins controller that was technically healthy and practically useless. The oldest job had been queued for 43 minutes. Two engineers were watching the same green progress bar on Slack. One of them suggested “just adding executors,” which is how Jenkins makes sure we remember it has a sense of humour.

We compared three paths for the services we build and deploy from GitHub: keep Jenkins on Kubernetes, move normal CI work to GitHub Actions, or run GitLab CI for the teams already using GitLab’s package and security features. This wasn’t a feature checklist exercise. We had 34 repositories, roughly 280 builds on a busy weekday, Java and Node services, Terraform plans, and one antique deployment job that talks to a vendor appliance through a jump host.

Our plain answer: GitHub Actions won for most application CI. Jenkins stayed for the strange jobs with network access, custom hardware, or years of scripted behaviour nobody wants to translate on a Thursday afternoon. GitLab CI is a decent choice if GitLab already owns your source control. We would not introduce it beside GitHub merely to replace Jenkins.

The Queue Problem Is Really a Worker Problem

Our Jenkins controller ran as a StatefulSet in EKS, with ephemeral Kubernetes agents created through the Jenkins Kubernetes plugin. On paper, this gave us elastic executors. In practice, the agent pod templates had grown into a 480-line shared library, node images were pulled too often, and a Docker-in-Docker sidecar occasionally sat in ContainerCreating while its sibling job waited.

The controller itself wasn’t the bottleneck. The queue was.

We had capped the ci-build node group at 12 m6i.large instances because it already cost enough during normal hours. Each Maven build requested 2 CPU and 4 GiB RAM. A few test suites asked for 6 GiB, Kubernetes packed them poorly, and cluster autoscaler took four to seven minutes to add nodes. By the time capacity appeared, another branch push had arrived.

GitHub Actions moved that waiting time somewhere else. Hosted runners started most Node and unit-test jobs in under a minute. The Windows jobs were less charming, but we only have four of those. GitLab’s shared runners behave similarly, though we have seen queue time vary more with GitLab.com’s shared capacity than we like for release work.

Here’s the blunt trade-off:

Criterion	Jenkins on Kubernetes	GitHub Actions	GitLab CI
Normal app build startup	2–8 minutes in our cluster	15–60 seconds	30–120 seconds on shared runners
Custom network access	Excellent	Requires self-hosted runner	Requires self-managed runner
Pipeline code	Powerful, often overgrown Groovy	YAML plus actions	YAML, generally cleaner than Jenkinsfiles
Plugin and action risk	Plugin upgrades can break controllers	Third-party actions need pinning	Templates and includes need review
Cost shape	EKS nodes, storage, admin time	Per-minute after included use	Per-minute or runner infrastructure
Debugging odd failures	Logs spread across controller, pod, cluster	Usually one run page	Usually one pipeline page
Our recommendation	Keep for exceptional jobs	Default for GitHub-hosted teams	Default only for GitLab-hosted teams

The surprise was that cost got worse before it got better. In the first month after moving 22 repositories, our GitHub Actions bill rose from about $180 in trial usage to $1,460. We had copied Jenkins behaviour too literally: every pull request ran integration tests, image builds, dependency scans, and a 19-minute Playwright suite.

Then we split checks by path, cancelled superseded runs, and put browser tests behind a merge-queue label. The next month landed at $690. Jenkins node spend also dropped by roughly $510, though our finance report made that connection about as obvious as a failed Helm rollback.

The Same Maven Build In Three Pipeline Files

We used the same service for the comparison: billing-api, a Spring Boot service with Maven tests, a container image, and a Helm deployment to staging. It has enough moving parts to expose weak spots, but it isn’t one of the cursed repositories.

Jenkins still gives us the most freedom. It also gives every team enough rope to build a second deployment system inside a shared library.

pipeline {
  agent {
    kubernetes {
      yaml """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: maven
    image: maven:3.9.9-eclipse-temurin-21
    command: ['cat']
    tty: true
"""
    }
  }

  stages {
    stage('Test') {
      steps {
        container('maven') {
          sh 'mvn -B -DskipITs verify'
        }
      }
    }
    stage('Image') {
      when { branch 'main' }
      steps {
        sh './ci/build-and-push.sh'
      }
    }
  }
}

The GitHub Actions version is shorter, and more of its behaviour is visible to a developer who has never had Jenkins administrator access. We pin third-party actions to commit SHAs in the real files; abbreviated tags below are easier to read but not what we permit in protected branches. GitHub’s own security hardening guidance is sensible here, particularly on action pinning and token permissions.

name: billing-api

on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read
  packages: write

jobs:
  test:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: "21"
          cache: maven
      - run: mvn -B -DskipITs verify

  image:
    needs: test
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4
      - run: ./ci/build-and-push.sh

GitLab CI is the least surprising YAML of the three for a conventional pipeline. Its rules syntax is better than the conditional knots we’ve seen in Jenkinsfile Groovy. GitLab’s runner executor documentation is worth reading before committing to shared versus Kubernetes runners; the operational split matters more than the syntax.

stages:
  - test
  - image

test:
  image: maven:3.9.9-eclipse-temurin-21
  stage: test
  cache:
    paths:
      - .m2/repository
  script:
    - mvn -B -DskipITs verify

image:
  stage: image
  image: docker:27
  services:
    - docker:27-dind
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
  script:
    - ./ci/build-and-push.sh

For a plain service, we’d choose Actions. Jenkins’ extra power only pays for itself when the job really needs it, not when someone wants an input step and a decorative Build With Parameters page.

Credentials Are Where Jenkins Shows Its Age

We spent more time on credentials than migration scripts.

Jenkins had 67 credentials in its store when we started. Twelve were still in use. Seven had names like prod-token-final2, which conveyed neither ownership nor comfort. A folder-scoped credential can be perfectly reasonable, but a Jenkins administrator can still reach a great deal of it, and plugin-specific credential bindings make audits tedious.

For GitHub Actions, we switched AWS deployments to OIDC and removed long-lived access keys from the pipeline path. The trust policy limits the role to our platform/billing-api repository and the main branch environment.

{
  "Condition": {
    "StringEquals": {
      "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
      "token.actions.githubusercontent.com:sub":
        "repo:acme-platform/billing-api:environment:production"
    }
  }
}

That was one of the few changes that felt immediately cleaner. The deploy job requests a short-lived token, AWS records the role session, and we stop pretending a secret rotated every 90 days is a happy secret. GitHub documents the OIDC flow for AWS well enough that we didn’t have to invent a wrapper.

GitLab supports the same general pattern with ID tokens. If your repositories live there, use them. We would avoid storing cloud keys in either CI product unless the target system has no federation option.

Jenkins can do OIDC too, via scripts, plugins, or an external secrets operator. We have it working for two jobs. The issue is consistency: every Jenkins library can choose its own route, and then the security review turns into archaeology.

We left one Jenkins credential in place: a client certificate required by that vendor appliance. Its API only accepts mutual TLS from a fixed private address, and its certificate rotation procedure arrives as a PDF once a year. Some systems do make their own case for retirement.

Could we move that appliance job to a self-hosted GitHub runner and shut Jenkins off entirely?

Self-Hosted Runners Recreate Part of the Jenkins Problem

Yes, we could. We tested it.

A self-hosted Actions runner in the private subnet could reach the appliance, mount the certificate from Secrets Manager, and execute the deployment script. The workflow itself was fine. The trouble began when we considered who would patch, drain, scale, and investigate those runners at 02:00.

Jenkins already had that machinery, imperfectly. Our agents were Kubernetes pods, isolated per build, and constrained with service accounts and network policies. A self-hosted runner can be ephemeral too, and GitHub recommends ephemeral runners for security-sensitive work. But we would still need an autoscaler, images, AMI or container patching, labels, runner registration, and enough logging to answer “what ran on this host?”

That’s Jenkins-shaped work wearing a GitHub badge.

We tried Actions Runner Controller for a week on a non-production cluster. It worked, but it added another controller, runner scale sets, GitHub App credentials, and operational ownership. Our platform team has five people, one of whom is on parental leave until October. We don’t have spare enthusiasm for another control plane.

This only held because the appliance job runs two or three times a week. If it ran 200 times a day, we would build the runner pool and accept the maintenance. At that volume, manual exceptions become their own incident category.

For now, Jenkins owns the private-network and hardware-adjacent jobs: appliance deploys, Android signing, and a load-test suite that pushes 25 Gbit/s through a lab VLAN. There are nine of them. Everything else is moving out.

Failed Builds Need Fewer Places To Look

Jenkins failures trained our team to open three tabs: build log, Kubernetes events, and Grafana. Four tabs if the shared library had changed. A red Jenkins job might mean a test failed, an agent could not schedule, the container registry throttled us, or the controller had lost a WebSocket connection to an agent.

GitHub Actions collapses most normal failures into one place. Logs are still noisy, especially when an action emits coloured furniture instead of useful output, but the run page has the checkout, cache, test, artifact, and deployment evidence together. GitLab offers a similar experience.

There’s a cost to that convenience. Hosted runners hide the host. When a GitHub ubuntu-24.04 image changed and our Playwright browser install broke, we had less ability to inspect the runner than we had with our own pods. Pinning the browser package fixed it, but the first failure message was just:

Error: browserType.launch: Executable doesn't exist

Our Jenkins setup had a different failure mode: the browser image was pinned for so long that it contained a six-month-old Chromium build. Predictability can quietly turn into neglect.

We also found that GitHub’s concurrency control stopped a lot of wasted work:

concurrency:
  group: pr-${{ github.event.pull_request.number }}
  cancel-in-progress: true

A developer pushing five commits in ten minutes no longer buys five copies of the same integration run. Jenkins can do this with milestone steps, abort plugins, or custom Groovy. We had all three patterns across repositories, naturally.

Marta, our Tuesday primary, preferred Jenkins logs because “at least the disaster is familiar.” She was right for the first fortnight. By week three, she stopped asking which agent pod had vanished.

Plugin Freedom Has a Maintenance Bill

Our Jenkins controller had 46 installed plugins. We removed 11 before the migration and still found dependencies on workflow APIs nobody had consciously selected. A plugin update is rarely exciting until it breaks a shared library and 34 repositories lose the ability to publish test reports.

We held Jenkins core at 2.452.3 for longer than we should have because an upgrade changed behaviour in the Kubernetes plugin. That wasn’t irresponsible exactly; it was a calculated refusal to spend a Friday proving that 120 pipelines still behaved. The calculation gets less defensible each quarter.

GitHub Actions has its own supply-chain problem. Marketplace actions are code from strangers with a reassuring logo. We allow approved actions, pin them by SHA, and use reusable workflows for container publishing and deployment. Teams can read those workflows, but they cannot casually add curl | bash to production release paths.

GitLab’s includes and templates offer a similar central-control model. We prefer it over Jenkins shared libraries because YAML is less capable of becoming a small programming language with opinions about classpaths. Some people love Jenkins Groovy. We have enough Java already.

There remains one unresolved argument in our team: whether reusable GitHub workflows are becoming shared libraries with worse local testing. They probably are. We’ve agreed to keep them thin—authentication, artifact naming, policy checks—and leave build commands in each repository. Ask us again after the next 20 migrations.

Pick This If Your Team Has These Constraints

Pick GitHub Actions if you have 3–30 engineers, repositories already hosted on GitHub, and mostly standard builds: tests, containers, Terraform plans, and cloud deployments. Start on hosted runners. Use OIDC for cloud access, protect environments, cancel duplicate pull-request runs, and resist adding self-hosted runners for one awkward service.

Pick Jenkins on Kubernetes if you have 15–80 engineers and a meaningful set of jobs that require private network paths, internal build hardware, custom operating systems, or long-established scripted release steps. Keep the controller small, make agents ephemeral, and assign ownership for every shared library. Do not keep Jenkins merely because the team knows where the blue button is.

Pick GitLab CI if source code, merge requests, container registry, and security scanning already live in GitLab. For a 10–50 person team, one product for those workflows is a sensible operational trade. We would choose GitLab CI over Jenkins in that setup, but not over GitHub Actions if moving source control is the hidden price.

For our five-person platform team, the split is 25 repositories on GitHub Actions, nine Jenkins jobs left behind, and a calendar reminder in November to see whether that vendor appliance has finally learned what an API is.