🛠️ Pipelines in the Wild #6
Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI
⚡ Byte Size Summary
- Implement the four-phase Expand/Contract pattern — nullable column, batched backfill, code pivot, then constraint — so schema changes never ship coupled to application code
- Understand why Argo CD's "rollback" can revert your container image but cannot touch a database schema, and why that gap is the actual thesis of zero-downtime deployment failures
- See the enforcement gap that let a smaller version of the same incident happen twice — and why process-only guardrails (PR review, runbooks) fail under deploy pressure
The Story
The cluster was OpenShift on AWS (ROSA-class multi-tenant), shared across several logistics sub-teams. The deploy was routine — Argo CD-driven rolling updates, "true zero-downtime" by design, database migrations bundled into the same release as the application code because that's how the pipeline had always worked.
A developer added a NOT NULL column to a high-traffic tracking table. No default value. It passed review. It shipped on a Friday afternoon.
The migration ran first and succeeded — that part of the pipeline worked exactly as designed. Then the rolling update started replacing pods, and the old-version pods, which had no idea the new column existed, tried to write to the table the way they always had. Every write failed. Checkout and tracking went dark in about 45 seconds. The rolling update itself stalled, because the new pods — which did know about the column — were crash-looping on readiness checks for an unrelated reason further down the deploy sequence.
The first response was the one that felt safest: hit rollback in Argo CD. It reverted the application image cleanly. It did nothing to the database. GitOps rollback and state rollback are not the same operation, and nobody in the room had internalized that distinction under pressure. The schema was still broken. The old code, now running again, still couldn't write to a table with a NOT NULL column it didn't know about.
Real recovery took four hours and involved a DBA forcing the lock. The stuck ALTER TABLE was holding an exclusive lock that the rollback had no way to release, so it had to be killed manually. Deployments were scaled to zero — a hard maintenance window, the thing zero-downtime deploys exist to avoid. Then the constraint itself came out:
ALTER TABLE tracking ALTER COLUMN new_feature_id DROP NOT NULL;
The system came back on the old code. Not a rollback in the sense anyone wanted to write down — an abandonment of the release, with the schema stripped back to something the old code could live with.
The blast radius didn't stop at the tracking table. Tracking pods, failing every write, spent CPU cycles reconnecting and retrying. That CPU pressure landed on shared compute nodes and starved adjacent microservices — teams with no relationship to the tracking database, on a cluster where compute was shared even though data stores weren't. A schema problem became a noisy-neighbor incident for services that never touched the broken table.
The Problem
Platform engineers running shared OpenShift clusters feel a specific version of this pain: zero-downtime deployment tooling — Argo CD, blue/green traffic splitting, rolling updates — was built to manage container lifecycle. None of it understands database schema state, and most teams don't notice the gap until a migration goes wrong mid-rollout.
The cost here wasn't abstract. It was 45 seconds of hard outage, a four-hour manual recovery involving a DBA killing a stuck lock, a full release abandoned back to the old code, and a cross-team incident on services that shared nothing but compute. The failure mode that made it worse than a simple bad deploy: Argo CD's rollback gave the appearance of a safety net. It reverted exactly what it manages — the application image — and nothing it doesn't — the schema. Trusting that rollback as a complete safety mechanism is what turned a bad migration into a four-hour incident instead of a thirty-second one.
Why Existing Approaches Fall Short
Coupling the migration to the code release. This is what the team was doing before the incident, and it's still the default in most CI/CD templates: migration and application deploy in the same Argo CD sync. It works exactly as long as every migration in every release is backward-compatible with the previous code version — which nobody enforces until the first time it isn't. Zero-Downtime Deployments with GitHub Actions and Feature Flags solved this coupling problem for application code years ago in this series. Nobody had solved it for schema — until this incident forced the question.
Trusting GitOps rollback as schema rollback. Argo CD reverts the manifests it manages. A database schema change made via a PreSync hook, a Job, or a manual ALTER TABLE is not tracked as part of the Argo CD-managed state in the same way — reverting the Application to a prior Git revision does not undo a completed ALTER TABLE. Teams that haven't hit this yet generally assume rollback is symmetric. It isn't.
Process-only migration review. PR review and a runbook checklist catch a NOT NULL without a default most of the time — which is exactly the problem. "Most of the time" is not an enforcement mechanism, and the honest gap section below covers what happened when this exact review process missed the same class of bug a second time.
The Architecture
The diagram makes one design decision visible: the migration path and the application release path share a pipeline trigger but never share a credential, a lock scope, or a rollback mechanism. In production GitOps, Argo CD's PreSync hook — not Sync Waves — is what enforces "schema before code" ordering. PreSync alone is sufficient when there's only one dependency to sequence (migration before app rollout). The companion lab validates the same Jobs, role split, and batched backfill on ROSA HCP via phase scripts; wire those Jobs as PreSync hooks when you put the pattern behind Argo CD.
Control plane: Argo CD (production) or run-phase.sh Jobs (lab), orchestrating expand → backfill → contract under a dedicated migrator ServiceAccount.
Data plane: Postgres (RDS in production; in-cluster Postgres in the ROSA HCP lab) and three database roles that gate access — db_ddl (Data Definition Language — schema changes: ALTER, DROP), db_dml (Data Manipulation Language — row-level writes: INSERT, UPDATE), and db_app (application runtime).
Blast radius: bounded by role separation. Even a fully compromised application pod, using the app/DML credential, has no path to ALTER TABLE. The DDL credential exists in exactly one place — the expand/contract Job's mounted secret — and nowhere else in the cluster.
OpenShift/Kubernetes specifics: the PreSync hook is an annotation on the Job manifest, not a Sync Wave. RBAC is a ServiceAccount with a local RoleBinding only — no ClusterRoleBinding, no secret get/list on the migrator Role (creds arrive via envFrom.secretRef at pod create time).
How It Works: Step by Step
Prerequisites
- ROSA HCP (or ARO) with
oclogged in — pattern validated on ROSA HCP workers - Postgres reachable from the cluster (lab: in-cluster Deployment; production: RDS / Azure DB in the same VPC)
- Three application-level DB roles:
db_ddl(expand/contract),db_dml(backfill),db_app(runtime) - Optional: External Secrets Operator + AWS Secrets Manager / Azure Key Vault
SecretStore(lab default uses fallback Secrets;USE_ESO=1swaps without changing Job specs) -
ocCLI matching cluster version
Step 1 — Expand (nullable column, decoupled from code)
-- EXPAND / DDL — nullable only. Safe under traffic with old code still writing.
ALTER TABLE tracking
ADD COLUMN IF NOT EXISTS new_feature_id INTEGER;
The column ships with no NOT NULL constraint and no default. This is the entire point of Expand — old code, which has never heard of this column, keeps working against the same table without modification. The companion lab demonstrates the same Expand shape on an orders table: nullable customer_first_name / customer_last_name, plus an empty order_items table.
Step 2 — Run the migration as a Job (PreSync in GitOps)
apiVersion: batch/v1
kind: Job
metadata:
name: db-migrate-expand
namespace: db-migration
labels:
app.kubernetes.io/name: db-migrate
app.kubernetes.io/component: expand
migration.phase: expand
annotations:
# Production GitOps wiring — lab applies this Job via run-phase.sh instead
argocd.argoproj.io/hook: PreSync
spec:
backoffLimit: 1
ttlSecondsAfterFinished: 86400
template:
metadata:
labels:
app.kubernetes.io/name: db-migrate
migration.phase: expand
spec:
serviceAccountName: db-migrator
restartPolicy: Never
containers:
- name: psql
image: postgres:16-alpine
env:
- name: PGOPTIONS
value: "-c search_path=app,public"
envFrom:
- secretRef:
name: db-credentials-ddl
volumeMounts:
- name: sql
mountPath: /migrations
readOnly: true
command:
- /bin/sh
- -ec
- |
echo "Phase=expand user=${PGUSER} host=${PGHOST}"
for f in $(ls /migrations/*.sql | sort); do
echo "==> $(basename "$f")"
psql -v ON_ERROR_STOP=1 -f "$f"
done
echo "Phase expand complete"
psql -c "SELECT version, phase, applied_at, applied_by FROM app.schema_migrations ORDER BY applied_at;"
volumes:
- name: sql
configMap:
name: migration-sql-expand
PreSync guarantees this Job completes before Argo CD proceeds to sync the application Deployment. No Sync Wave ordering needed for a single migration-before-code dependency. Expand SQL is idempotent (ADD COLUMN IF NOT EXISTS) — if Expand fails mid-script, re-run the phase.
Bookkeeping here is inline: each expand/contract SQL file ends with its own INSERT INTO schema_migrations statement, since the whole file runs exactly once. Backfill can't do that — a single backfill file runs many times in a loop — so it uses a separate bookkeeping call instead, covered in Step 3.
Step 3 — Batched backfill via Job (DML role)
Backfill uses its own template (openshift/jobs/backfill-job.yaml.tpl), not the shared expand/contract template from Step 2. Unlike a single-shot schema change, backfill has to move through a large table in bounded batches without holding a long lock — which is the direct lesson from the incident: an unbatched UPDATE across every row takes an exclusive lock proportional to table size and holds it for the whole statement. On a small table that's invisible. On a high-traffic production table, it's the same failure mode that caused the original outage.
Each backfill SQL file updates at most :batch_size rows per execution and reports how many it touched:
-- migrations/backfill/V003__backfill_split_names.sql
SET search_path TO app, public;
WITH batch AS (
SELECT id
FROM orders
WHERE customer_first_name IS NULL OR customer_last_name IS NULL
ORDER BY id
LIMIT :batch_size
),
updated AS (
UPDATE orders o
SET
customer_first_name = COALESCE(
o.customer_first_name,
NULLIF(split_part(trim(o.customer_name), ' ', 1), '')
),
customer_last_name = COALESCE(
o.customer_last_name,
NULLIF(
CASE
WHEN position(' ' IN trim(o.customer_name)) = 0 THEN ''
ELSE substring(trim(o.customer_name) FROM position(' ' IN trim(o.customer_name)) + 1)
END,
''
)
)
FROM batch
WHERE o.id = batch.id
RETURNING o.id
)
SELECT count(*) FROM updated;
The Job's shell loop calls this file repeatedly via psql -tA, reads the printed row count back into the shell, sleeps briefly between batches, and stops the moment a batch returns 0:
run_batched() {
local version="$1" phase="$2" file="$3" total=0 n
while :; do
n="$(psql -tA -v batch_size="${BATCH_SIZE}" -f "$file" | tail -n1)"
total=$((total + n))
if [ "$n" -eq 0 ]; then
break
fi
sleep "${BATCH_PAUSE_SECONDS}"
done
psql -v version="${version}" -v phase="${phase}" -f /scripts/mark-migration-applied.sql
}
run_batched "V003" "backfill" /migrations/V003__backfill_split_names.sql
run_batched "V004" "backfill" /migrations/V004__backfill_order_items.sql
BATCH_SIZE=500, BATCH_PAUSE_SECONDS=0.25 — small batches, brief lock, released between each one, rather than one lock held for the duration of the entire backfill.
Progress is tracked in a schema_migrations table, written once per file — after its loop completes, not once per batch:
-- scripts/mark-migration-applied.sql
INSERT INTO schema_migrations (version, phase)
VALUES (:'version', :'phase')
ON CONFLICT (version) DO NOTHING;
This is why a re-run after a partial failure doesn't reprocess files that already fully completed — the bookkeeping insert only happens once the loop for that file has already returned 0.
Backfill runs as db_dml — it writes data, it never touches schema, so it never mounts the DDL secret.
Step 4 — Code pivot (N+1), only after 100% backfill confirmed
New code deploys with defensive null-handling regardless of backfill status — a second line of defense, not a substitute for confirming backfill completion first. In the lab this is the dual-write → dual-read → cutover sequence (WRITE_MODE / READ_MODE on the app Deployment) before Contract is allowed to run.
Step 5 — Contract (after old code is fully drained)
Contract does two things, not one: it drops the legacy columns the migration was moving away from, and only then hardens the replacement columns with NOT NULL. Doing both in the same phase — rather than dropping legacy columns separately — is deliberate: there's no reason to carry dead columns forward once nothing reads them.
-- migrations/contract/V005__contract_drop_legacy.sql
ALTER TABLE orders DROP COLUMN IF EXISTS customer_name;
ALTER TABLE orders DROP COLUMN IF EXISTS items;
-- Harden new columns once legacy is gone
ALTER TABLE orders
ALTER COLUMN customer_first_name SET NOT NULL,
ALTER COLUMN customer_last_name SET NOT NULL;
By this point every row has a value, so the SET NOT NULL validates without holding an exclusive lock across a long table scan under load. The DROP COLUMN calls are the less reversible half of this phase — there's no cheap rollback for a dropped column, only restore from backup or re-add it and re-backfill from scratch. Never contract until metrics show zero legacy reads.
Security Considerations
DB-level role separation is the actual control, not a suggestion. db_ddl (schema owner, expand/contract) is used exclusively by DDL Jobs. db_dml (SELECT/INSERT/UPDATE/DELETE only) runs backfill. db_app is what application pods use. Even a fully compromised app pod has zero path to ALTER TABLE — enforced at the database role level, not by convention.
Secret separation via ESO (or named Secrets). Dual/triple credentials live in AWS Secrets Manager (or Azure Key Vault on ARO), synced into OpenShift via External Secrets Operator — the same multi-cloud ESO pattern covered in Secrets Management Across Multi-Cloud Pipelines — or, in the lab path, three explicitly named Secrets (db-credentials-ddl, db-credentials-dml, db-credentials-app). The DDL secret is mounted only into expand/contract Jobs. Application Deployments never reference it. The lab's USE_ESO=0 fallback path carries the same RBAC scoping as the ESO path — role separation is enforced by which Secret each Job or Deployment mounts, not by which secret-management backend supplies it. Skipping ESO for local testing doesn't loosen the boundary.
RBAC scoped to independent audit identity. db-migrator exists so migration actions have their own traceable identity in the audit log, separate from the application's service account. It's bound to local RoleBindings only — Jobs, pods/logs, ConfigMaps in-namespace. No secret API read on the Role (injection is envFrom at create time), no ClusterRoleBinding, no cross-namespace access.
Tradeoffs
What you gain / what you give up
Decoupling migration from code deploy buys you the ability to ship a schema change independently of a release — and to have old code and new code coexist against the same schema without either one breaking. What you give up is deployment simplicity: a single feature now takes four release cycles to fully land (expand, backfill, pivot, contract) instead of one. For a small, fast-moving team, that overhead can feel disproportionate to the risk it prevents — until the first incident makes the tradeoff obvious.
Enforcement remains an open gap, not a solved problem. The Expand/Contract pattern is a technical answer. It does not, by itself, stop someone from writing a NOT NULL column with no default into an Expand script. Guardrail enforcement here has been process-only — PR review and a runbook — not automated. That's the honest limitation, and the next section covers what happened because of it.
What I'd Do Differently
The guardrail enforcement was process-only, and that gap recurred in production: a rushed developer later slipped a NOT NULL column into a Phase 1 script anyway, PR review missed it under the same time pressure that caused the original incident, and it triggered a smaller repeat incident.
The fix isn't more review — it's automation. I'd stop relying on human discipline to catch an Expand script with a hard constraint in it, and instead force DB schema linting into CI, so a NOT NULL without a default in an Expand migration fails the pipeline before it can be approved by a human at all. This isn't a hypothetical improvement — it's a lesson earned from watching the same category of mistake happen twice.
What Breaks at Scale
At around 40+ microservices and roughly a dozen teams, the Expand/Contract discipline itself became the bottleneck rather than the database. A single feature deployment turned into a three-week, multi-release orchestration effort — expand, backfill, pivot, contract, each with its own review and release window. The failure mode at that scale wasn't technical. It was cognitive overhead and pipeline velocity friction: teams start looking for ways to skip phases under deadline pressure, which is exactly the condition that produced the repeat incident described above. It's the same category of discipline-at-scale problem covered from the infrastructure-state side in Managed OpenShift, Lost State, and Daily Drift Checks — more moving parts don't just add work, they add places for the process itself to quietly fail.
Quick Recap
- GitOps rollback is not schema rollback — Argo CD reverts what it manages (the application manifest); a database schema change is a separate operation with its own recovery path, and conflating the two turned a fixable bug into a four-hour incident
-
Expand → batched backfill → pivot → contract — nullable first, DML-only backfill in ~500-row batches, drop legacy columns and constrain only after old code is gone;
PreSyncwires migration-before-code ordering when you run this under Argo CD - Process-only enforcement fails under deploy pressure — the same class of bug (a hard constraint with no default) got through PR review twice; DB schema linting in CI is the fix, not more review discipline
GitHub Repo
Validated on ROSA HCP (Expand/Contract Jobs, DDL/DML/app role split, batched backfill, dedicated db-migrator SA). The lab uses an orders table as a stand-in for the incident schema — same pattern, not a literal reproduction of the tracking table. Argo CD PreSync annotations are the production GitOps wiring for those Jobs; the lab drives phases with ./scripts/run-phase.sh.
What's Next?
Pipelines in the Wild #7 isn't scheduled yet. If you've hit a production war story worth covering — a specific incident, a specific fix, a specific thing you'd do differently — open an issue against the lab repo or flag it directly. That's genuinely where the next one starts.
Written by Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI
Top comments (0)