Honey Kiran

Posted on Oct 26 Originally published at honeykiran.substack.com on Oct 17

Your Deploy Passed Every Check. Half the App Was Missing.

#devops #cicd #sre #kubernetes

Somebody on r/devops asked the question plainly this week: “What does your deployment actually verify?” Sit with it for a second, because the honest answer is uncomfortable. Most deployments verify that a process started. They do not verify that the application works — and they never even look at the data the application stands on.

The scene is invented — no names, no real incident — but you have lived a version of it. Friday, late afternoon. The pipeline is green, the new pods roll out, every probe passes, and the release message goes out with a rocket emoji. Twenty minutes later support forwards a user complaint: the signup form’s country dropdown is empty. You check the endpoint yourself. It answers HTTP 200 — with an empty list, []. The app reads its lookup data — countries, currencies, price plans, the boring reference tables nobody thinks about — from the database, and in this environment those tables were never seeded. The process was perfectly alive. It was singing to its health check while the product stood next to it, missing a floor.

Layer one: the process is breathing

Start with what your current checks actually are. A probe is the kubelet — the agent that runs on every Kubernetes node — poking your container on a timer to see if it flinches. Kubernetes gives you three probes, and each answers exactly one narrow question:

Startup probe — “has the app finished starting?” Until it succeeds once, the other two probes stay silent, so a slow starter does not get killed by its own liveness probe for the crime of booting slowly.
Liveness probe — “is this process stuck?” Fail it past the configured failureThreshold and the kubelet restarts the container. That is the whole job: catch a deadlock, restart, move on.
Readiness probe — “should this pod get traffic right now?” Fail it and the pod’s IP is removed from the Service’s endpoints, so requests stop arriving. The container keeps running, unwatched and unharmed, until it is ready again.

Now the part nobody reads. An HTTP probe calls any status from 200 to 399 a success. A TCP probe succeeds if the port merely opens — Kubernetes’ own docs note that if the other side closes the connection immediately, “this counts as healthy.” So a green probe proves one thing: something answered. An empty body, a redirect to a maintenance page, a payload full of nothing — all green. The probe was never designed to ask whether the answer was useful.

A health check proves the process is breathing. It has never once proved the app can work.

And one misuse turns this small tool into a weapon. Wire a dependency check into liveness — the classic /healthz endpoint that pings the database and returns 500 when the database is unhappy. Now let the database have one bad minute. Every pod checks, every pod fails, every pod restarts at once — and fifty fresh processes slam the already-sick database with new connections, fail again, restart again. Kubernetes’ documentation warns in plain words that badly built liveness probes cause cascading failures. A dependency check belongs, at most, in readiness, where failing only pauses traffic. And even readiness is a traffic gate. It is not a functional test, and it never will be.

Layer two: the app can do its job

Layer two is proof the app can execute its actual work. The tool is the smoke test: a short scripted walk through the two or three user journeys that pay the bills — sign up, search, check out, whatever your version of those is — run immediately after the deploy. Short matters. A gate that takes an hour will be bypassed by Friday; a gate that takes two minutes becomes a habit.

The cardinal sin of smoke testing is asserting on the status code. A 200 can wrap an empty list, an error message wearing a JSON costume, or yesterday’s cached answer. Assert on the payload. If the journey needs the countries list, the test demands a full list with the entries it expects — not a 200 and a shrug. If search matters, the test searches for something it knows exists and demands priced, well-formed results back. Here is the shape of it:

#!/usr/bin/env bash
# smoke.sh - the journeys that pay the bills, asserted on payload, not status
set -euo pipefail
BASE="https://app.example.com"

fail() { echo "SMOKE FAIL: $1" >&2; exit 1; }

# Journey 1: the reference data the signup form needs
body=$(curl -fsS "$BASE/api/v1/countries") || fail "countries endpoint did not answer 2xx"
count=$(jq 'length' <<<"$body")
["$count" -gt 100] || fail "countries list has $count entries - reference data missing?"
jq -e '.[] | select(.code == "IN" and .name == "India")' <<<"$body" >/dev/null \
  || fail "expected entry missing from the countries payload"

# Journey 2: search returns real, priced results
body=$(curl -fsS "$BASE/api/v1/search?q=shoes") || fail "search did not answer 2xx"
jq -e '.results | length >= 3' <<<"$body" >/dev/null \
  || fail "search returned fewer results than a working index should"
jq -e '.results | all(.price > 0 and (.id | type == "number"))' <<<"$body" >/dev/null \
  || fail "search results contain unpriced or malformed items"

echo "SMOKE OK: the journeys answered with the payloads users actually need"

Then do not let those journeys sit idle between deploys. Run the same flows on a schedule, around the clock, as synthetic checks — fake users who walk the real routes whether or not real users are awake. AWS built this into CloudWatch Synthetics, whose canaries “follow the same routes and perform the same actions as a customer” precisely so you can verify the experience “even when you don’t have any customer traffic”; Google Cloud’s synthetic monitors and uptime checks do the same job, and the uptime checks can validate the response content, not only the status. Layer two, run continuously, is how you hear about the 4 AM breakage from your own tooling instead of from the first customer of the day.

Layer three: the data the app stands on

Now the layer almost nobody verifies — the one that was missing in the scene above. Code rolls back in seconds. Data is sticky, heavy, and silent: when it is absent or wrong, the app does not crash; it politely misbehaves, which is worse. Four things belong to this layer:

Reference and seed data — the lookup tables the code reads to make decisions: countries, currencies, tax rules, permission sets, plan tiers. In a developer’s laptop environment these get seeded by hand and forgotten. In a fresh environment they are simply not there. Seeding must be part of the deploy — versioned, automatic, and verified — never a script somebody ran once in March.
Migrations verified forward — schema changes split into backward-compatible steps, the expand–contract pattern: expand (add the new column or table alongside the old one), migrate (backfill, dual-write, switch reads), contract (drop the old shape in a later deploy). During any rollout, old and new pods run side by side, so the schema must serve both. The payoff: rolling back the app never requires rolling back the schema — the scariest rollback there is, deleted from the incident.
Schema and data checks after migrating — a migration tool exiting zero is a claim, not a proof. Query the database itself: does the new column exist, are the row counts what the migration promised, are the critical lookups returning rows? If the backfill was supposed to leave zero NULLs behind, count the NULLs.
Config, secrets and flags as a gate — the environment variables, mounted secrets and feature-flag states this release expects should be verified present before traffic arrives. A missing secret discovered by the first user is not a user discovery. It is a deploy that verified nothing.

The canary is a witness, not a decoration

Plenty of teams run canaries — a small slice of traffic on the new version before everyone gets it — as a ritual: shift 5%, wait ten minutes with nobody watching, promote anyway. Google’s SRE books describe the real thing: canarying is a partial, time-limited deployment that gets judged. The canary is compared against the control — the current version, same traffic, same conditions — and if the canary’s error rate drifts too far from the control’s, you pause and roll back. If the two behave alike, you proceed. The canary exists to testify. Give it no power to fail the deploy and it is just a slower rollout with extra dashboards.

The tooling can hold the gavel. Argo Rollouts’ AnalysisTemplates query your real metrics on an interval — request success rate from Prometheus, say — against a condition you set, and when the measurements fail past the limit you allow, the rollout aborts and the canary is rolled back on its own. Flagger goes one step earlier: its pre-rollout webhooks run your smoke tests against the canary before it receives a single user request, and a failing test means the traffic never shifts at all. Either way the principle is identical: the verdict comes from evidence — error rates, latency, payload assertions — not from the colour of the pipeline.

Gate it in the pipeline

Verification that lives in a wiki page is a wish. Each layer has to become a step that can stop the release. In a GitOps setup the bones already exist: Argo CD runs annotated Jobs as hooks in its sync phases. A migration Job marked PreSync runs before any manifest is applied — the Argo CD docs use exactly this as their example — and if any PreSync hook fails, the whole sync stops and is marked failed. The old version keeps serving. Nothing broken reaches a user.

# The migration runs BEFORE the new code - and can stop it.
# Argo CD: a failed PreSync hook fails the whole sync.
apiVersion: batch/v1
kind: Job
metadata:
  generateName: db-migrate-
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  backoffLimit: 0 # fail loudly, do not quietly retry a bad migration
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: registry.example.com/shop/api:2.14.0
          command: ["./migrate", "up"] # expand step only; contract ships later

After the migration, gate on the data itself — schema present, rows present, from inside the cluster:

#!/usr/bin/env bash
# data-gate.sh - schema is a claim; rows are proof (adjust names to your stack)
set -euo pipefail

rows=$(kubectl exec -n shop deploy/api -- \
  psql "$DATABASE_URL" -tAc "SELECT count(*) FROM countries")
["$rows" -gt 100] || { echo "GATE FAIL: countries holds $rows rows"; exit 1; }

col=$(kubectl exec -n shop deploy/api -- \
  psql "$DATABASE_URL" -tAc \
  "SELECT count(*) FROM information_schema.columns
    WHERE table_name = 'orders' AND column_name = 'currency_code'")
["$col" -eq 1] || { echo "GATE FAIL: orders.currency_code does not exist"; exit 1; }

kubectl get secret api-secrets -n shop >/dev/null \
  || { echo "GATE FAIL: api-secrets not present in this namespace"; exit 1; }

echo "GATE OK: schema verified, reference rows present, secrets mounted"

And the whole thing reads as one small script the pipeline runs after the rollout — the deploy is not done until this says so:

#!/usr/bin/env bash
# verify-deploy.sh - run AFTER rollout, BEFORE anyone says "deployed"
set -euo pipefail
NS="shop"

echo "Layer 1 - process: pods up, Ready, not restart-looping"
kubectl rollout status deployment/api -n "$NS" --timeout=180s
restarts=$(kubectl get pods -n "$NS" -l app=api \
  -o jsonpath='{.items[*].status.containerStatuses[*].restartCount}' \
  | tr ' ' '\n' | awk '{s+=$1} END {print s+0}')
["$restarts" -eq 0] || { echo "FAIL: $restarts container restarts since rollout"; exit 1; }

echo "Layer 2 - function: the journeys, asserted on payload"
./smoke.sh

echo "Layer 3 - data: schema, reference rows, secrets"
./data-gate.sh

echo "DEPLOY VERIFIED: process alive, journeys working, data present"

Staging warning: before you trust this script in production, prove it can fail. In staging, delete the reference rows on purpose and run the pipeline. If it still goes green, your verification is an ornament — fix it there, where the lesson is cheap. And smoke tests that create data (test users, test orders) need a dedicated test account and a cleanup step; a production database slowly filling with test_user_99 is its own small incident, scheduled for later.

Prove it before you call it deployed

Liveness checks the process, nothing else — no database pings, no dependency calls; a dependency’s bad minute must never restart your fleet.
Readiness is treated as a traffic gate — it decides who gets requests, and nobody on the team calls it a functional test.
The smoke test walks the journeys that pay the bills — sign up, search, buy — immediately after every deploy.
Assertions read the payload — expected entries, plausible counts, priced results. A bare 200 fails nothing and proves nothing, so it is not accepted as evidence.
The same journeys run on a schedule — synthetic checks keep verifying at 4 AM, when no real user is around to be your monitoring.
Migrations ran as a gate, forward and compatible — expand before code, contract later; app rollback never needs a schema rollback.
The data was queried, not assumed — new schema objects exist, backfill counts match, reference tables hold their rows in this environment.
Config, secrets and flags were verified present — in the states this release expects, before the first real request arrived.
The canary was judged against the control — error rate and latency compared, with automatic abort and rollback armed, not a human glancing at a graph.
The verification has been seen failing — somebody broke staging on purpose and watched every gate turn red. A check that has never failed has never been tested.

The honest sentence

Listen to how the word gets used in your next incident review. “The deployment succeeded, but the app was broken” is a sentence that should not survive the meeting. If the app was broken, the deployment did not succeed — a container started, which is a much smaller event, and it deserves its smaller word.

So borrow the honest sentence for your release notes: deployed means the process is alive, the journeys return what users need, and the data the app stands on is present and correct — proven, in that order, by checks that can each say no. A health check proves the process is breathing. Only verification proves the app arrived.

Honey Kiran · DevOps & Platform Engineering

DevOps, SRE, Terraform, and platform engineering — explained the practical way.

What does your deployment actually verify today — which of the three layers would yours fail? Tell me in the comments; the layer-three stories are the ones worth hearing.

Examples are illustrative — validate in your own environment before running in production.

Disclaimer

This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company's intentions, policies, or internal decisions. Where I do not know something, the post says so.
I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.
Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.
Nothing in this post is professional, financial, legal, or security advice. Examples are illustrative — validate in your own environment before running anything in production.
Corrections are welcome: if a company named here publishes an official statement, or a detail above is wrong, I will update this post and note the correction. The goal is an accurate public record of my experience, not a verdict on anyone.

DEV Community