DevOps Questions After We Broke The Release Handshake

Answers from the incident where every dashboard looked politely wrong.

The release-api deployment had already been marked complete when the invoice page began returning 503s. The new container was serving traffic, the PostgreSQL migration had committed, and the feature flag was on. A NetworkPolicy added in another repository prevented the new pod from reaching tax-rate-cache.

The application team saw errors, the database team saw a clean migration, and Platform saw green nodes. By the time we put all three facts in one incident channel, 63 deployment messages had buried the one that mattered.

“Is This Actually A DevOps Failure Or Just One Bad Deploy?”

It was a DevOps failure because four teams completed valid local work and nobody owned the release handoff between them.

Calling it “just a bad deploy” would have been convenient. We could have fixed the policy, replayed the release, written a short incident note, and carried on pretending that a green Argo CD application means a service is ready for users. The pod was healthy according to Kubernetes. It was also unable to call a dependency required to render an invoice. Both things can be true, which is why a deployment status alone is a fairly poor witness.

Our old release process had hidden contracts in too many places:

• The service repository declared its image and Helm values.
• The infrastructure repository held network rules.
• Database migrations ran from a separate GitHub Actions workflow.
• Feature flags lived in LaunchDarkly, owned by whoever had last touched the feature.
• The runbook lived in Confluence, where it had last been edited in February.

We’ve started putting the release dependencies in the service repository, close to the code that needs them. It is not a clever system. It is a file that a human can read during an incident and a pipeline can check before promotion.

release:
  service: release-api
  requires:
    - dependency: tax-rate-cache
      namespace: finance
      port: 8080
      network_policy: allow-release-api-to-tax-rate-cache
    - migration: 2026_08_17_add_tax_region
      minimum_version: "4.12.0"
  feature_flags:
    - invoice_tax_v2
  smoke_test:
    command: "./scripts/check-invoice-preview.sh"
    timeout_seconds: 90

The pipeline does not yet validate every field. It checks that the named NetworkPolicy exists in the target cluster and runs the smoke test after the migration. That alone would have caught Monday’s problem before 11% of invoice-page requests failed.

We dislike using “DevOps” as a department name. In this case, it means the work of making handoffs visible before production discovers them for us.

“Do Developers Really Need Production Access?”

Yes, but they need narrow production access tied to the service they own, not a shared kubectl credential passed around like an office spare key.

For too long, our answer was to give engineers log access and tell them to ask Platform for anything else. That worked while we had six services and one cluster. We now have 34 services, three production clusters, and an on-call rota where the person holding the pager may know the application code better than the person who built the cluster.

The useful access is usually boring:

• Read logs, events, and deployment state in the team namespace.
• View configured secrets by name, never by value.
• Restart a failed deployment through a recorded workflow.
• Fetch an approved diagnostic profile during an incident.
• Read relevant cloud metrics and traces.

The access we do not hand out includes cluster-wide secret reads, direct production database writes, and unrestricted exec into pods. There are rare reasons for each. Those reasons deserve an incident ticket and a second human looking at the command.

Our Kubernetes groups now map to service ownership. The catalog team cannot casually inspect identity, and Platform cannot claim every namespace is somebody else’s concern.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: catalog-oncall
  namespace: catalog-prod
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log", "events"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets"]
    verbs: ["get", "list", "watch", "patch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: []

The empty pods/exec permission is deliberate. We had one incident in May where an engineer fixed a live configuration file inside a container, then forgot it existed. The next rollout erased it, as containers are designed to do. We spent forty minutes rediscovering the same failure with more coffee involved.

Kubernetes has plenty of sharp edges here, so we keep the policies close to the upstream RBAC model rather than inventing an internal permissions language.

“Why Can’t Platform Just Own Every Deployment?”

Platform can own the deployment path; it cannot own the operational knowledge of every service.

We tested the “Platform approves production” model last autumn. It looked safe on a process diagram. In practice, it put two platform engineers between every product team and a release, including changes that only altered invoice wording or a CSS bundle. The queue grew, people started sending urgent messages, and approval became a ritual performed after somebody had already decided to deploy.

That is gatekeeping with better YAML.

The service team now owns whether its code should go live. Platform owns the tooling that makes the decision visible, repeatable, and reversible: the GitHub Actions templates, the Argo CD projects, the cluster policies, and the audit trail. Security owns the controls that must be true before the workflow can issue production credentials. Those boundaries are written down because informal agreement lasts exactly until the person who remembers it is on holiday.

We use CODEOWNERS for files that change shared deployment behavior. A service team can change its replica count or an application environment variable. A change to a shared ingress class, an organisation-level action, or a production NetworkPolicy requires Platform review.

# .github/CODEOWNERS
/.github/workflows/reusable-deploy.yml  @platform-engineering
/platform/network-policies/            @platform-engineering @security-engineering
/services/catalog/                     @catalog-team
/services/release-api/                 @billing-team

GitHub documents the awkward parts of CODEOWNERS matching and branch protection, including the fact that a file can have one visible owner line while several teams assume they own it. We found two of those in the first review.

The rule is not “Platform stays out of releases.” We still join risky database changes, regional failovers, and anything touching shared ingress. We simply refuse to become the human merge queue for routine service work.

“What Does ‘You Build It, You Run It’ Mean On Pager?”

It means the team that changes a service takes first call for its user-facing failures, with Platform covering failures in the shared runtime.

We had to make that sentence more precise than it sounds. “Run it” had become a polite way to tell application engineers that they owned every alert, including node pressure, expired cluster certificates, and a broken Fluent Bit daemonset. Nobody learns much from a pager at 03:00 that says “the whole platform is vaguely unhappy.”

Our current split is based on the failing layer. If catalog-api returns 5xx responses because of a bad release, Catalog gets paged. If every namespace loses DNS resolution, Platform gets paged. If the application cannot reach Redis because its own connection pool is exhausted, Catalog gets paged; if the Redis service itself is unavailable, the data platform rotation gets paged and Catalog gets an incident notification.

That sounds obvious after writing it down. It was not obvious in PagerDuty, where 19 services had a generic escalation policy called production-critical.

We changed the alert annotation format so it carries an owner, a runbook URL, and a statement of what user action is failing. The last field has reduced some thoroughly unhelpful alerts.

annotations:
  owner_team: catalog
  runbook: https://ops.example.internal/runbooks/catalog-checkout-errors
  user_impact: "Customers cannot add an item to an existing order"
  dashboard: https://grafana.example.internal/d/catalog-api

Mina, who has been on our platform rota longer than the current Grafana folder structure, objected to putting ownership in annotations. Her point was reasonable: labels rot. She was right, so the alert rule now validates the owner against our service catalog during CI. We still have stale runbooks, but now they fail a check instead of waiting quietly for a bad night.

For alert design, we keep returning to the Google SRE Workbook guidance on paging: page for urgent user impact, not every noisy internal symptom. It has saved us from several clever alerts that nobody could act on.

“Are DORA Metrics Going To Become Another Scorecard?”

They will if we publish team rankings, so we do not.

We track deployment frequency, lead time, change failure rate, and time to restore because they help us locate friction in the delivery system. The DORA research is useful as a vocabulary. It does not grant permission to declare that the team with fewer deployments is failing.

Our billing team deploys release-api three or four times a week. The reporting team deploys its batch export service about twice a month, often after waiting for a partner’s sample file. Comparing those numbers is managerial astrology.

Instead, we review metric changes against specific events. In July, median lead time for Catalog rose from 7 hours to 31 hours. The cause was not developer speed. A required image scan had begun waiting behind the same two self-hosted runners that handled integration tests. We added two runners, separated scan jobs from test jobs, and lead time dropped the following week.

Change failure rate exposed a less comfortable issue in Billing. Their rate was 18% over six weeks, mostly because a database migration and application release were approved separately. The failed release-api deployment belongs in that number. We are not using it to tell the team to make fewer changes; we are using it to fund the work of joining the changes safely.

We show rolling twelve-week trends and attach incident links to failed changes. Nobody gets a traffic-light score. If a metric appears in a performance review, people will optimise the number and hide the problem. We have enough experience with that particular species of spreadsheet.

“Should We Put Argo Rollouts In Front Of Every Service?”

We don’t know yet, honestly.

We have Argo Rollouts running for three HTTP services: catalog-api, release-api, and notifications-webhook. The canary step sends 10% of traffic to the new ReplicaSet, waits five minutes, then checks request error rate and p95 latency. If either crosses the configured threshold, it aborts.

strategy:
  canary:
    steps:
      - setWeight: 10
      - pause:
          duration: 5m
      - analysis:
          templates:
            - templateName: checkout-http-health
      - setWeight: 50
      - pause:
          duration: 10m
      - setWeight: 100

The release-api incident would probably have failed its canary analysis. “Probably” matters. Our staging traffic is too thin to produce a meaningful request-error signal, and production canaries only help when the issue appears in the first slice of traffic. A missing network permission would have done so. A quarterly invoice batch failure would not.

We have not tried it past 40 nodes, and our production clusters have 27, 31, and 38 worker nodes. The controller overhead looks modest in our tests. That is not evidence for what happens after the acquisition team’s services arrive, assuming that deal survives the budget meeting.

We also do not want rollout machinery around every workload. A nightly reconciliation Job does not need traffic shifting. Neither does a CronJob that fetches exchange rates at 04:15. For these, a failed run needs a clear alert and an easy rollback of the image tag. Adding canary objects would make the repository busier without reducing the relevant risk.

Argo’s analysis templates are useful because they force us to state what “healthy” means. The hard part is choosing a metric that reflects users rather than deployment optimism.

“How Will We Know The New Release Rules Are Working?”

We’ll know when a service team can explain a failed production release from one pull request, one deployment record, and one incident timeline.

That is our test for the next six weeks. Each production deployment must link its commit, container digest, migration version, feature flags, smoke-test result, and rollback action. We are adding those fields to the release event emitted into OpenTelemetry, then building the incident view from that event instead of asking people to paste screenshots into Slack.

The first version will be ugly. It only held together this week because the billing team already had consistent Git tags and the migration workflow happened to emit a version number. Catalog does not yet do either. We are fixing the release metadata there before we announce a company-wide standard and discover that standards are easiest when somebody else has done the plumbing.

We do not need a 64th deployment message.

The remaining argument is whether a NetworkPolicy change should block a service release until its owning team approves it, or whether the platform team should approve it as shared infrastructure. We have put that decision on Thursday’s architecture agenda, where it will receive the traditional amount of adult supervision.