DEV Community: Petr Petrenko

Your Database Just Died. Why Is Everything Still Running?

Petr Petrenko — Mon, 15 Jun 2026 05:50:46 +0000

It's 3 AM. Your PostgreSQL pod crashes. On-call fires. The engineer wakes up, checks the dashboard, and spends 20 minutes figuring out why the payments service is throwing 500s at 100% error rate before they realize — database is down, but payments is still running, hammering connection pools, logging thousands of errors per second, triggering cascading alerts.

They scale payments to zero. Problem stops. Recovery begins.

That 20 minutes was avoidable. klink would have scaled payments to zero automatically — 30 seconds after the database went down.

The Problem Nobody Talks About

Kubernetes is excellent at keeping individual services running. Liveness probes, restart policies, resource limits — it's all there. But Kubernetes has no concept of relationships between workloads.

When your database fails, Kubernetes does exactly what you told it to: keeps the dependent services running. Those services now do nothing useful — they just generate noise, consume resources, and make your incident harder to debug.

This is the gap klink fills.

What klink Does

klink introduces a new primitive: WorkloadDependency. You declare that service B depends on service A. klink watches. When A goes unhealthy, klink scales B to zero. When A recovers, klink restores B automatically.

apiVersion: deps.klink.dev/v1alpha1
kind: WorkloadDependency
metadata:
  name: payments-needs-database
  namespace: production
spec:
  dependent:
    kind: Deployment
    name: payments-service

  dependsOn:
    - kind: Deployment
      name: postgresql
      condition:
        minReadyPercent: 80    # healthy if ≥80% pods ready
        window: 30s            # ignore transient restarts
        recoveryWindow: 60s    # wait for stability before restoring

  onDegraded:
    action: ScaleToZero

  mode: strict

That's it. No code changes. No sidecars. No complex configuration.

How It Works

The hysteresis window is critical. Without it, a single pod restart would cascade a shutdown. With window: 30s, klink ignores transient failures — only sustained outages trigger action.

Enforcement Modes

Different situations call for different behavior. klink has four modes:

Mode	What it does	When to use
`strict`	Scales to 0 on failure. Reverts manual scale-ups within 15s.	Production services where cascade is non-negotiable
`soft`	Scales to 0 once. Respects manual overrides.	Services where operators need flexibility during incidents
`gate`	Doesn't scale down. Blocks scale-up via admission webhook.	Preventing HPA from scaling up while dependency is down
`observe`	Logs what it would do. Takes no action.	Safe onboarding — see what klink would do before enabling it

Start with observe mode. Apply klink to your existing services, watch the logs for a week, and only switch to strict or soft once you're confident.

spec:
  mode: observe  # "would scale payments to 0 — dependency unhealthy"

The Mutual Dependency Problem

What happens when A depends on B and B depends on A?

Naive implementations deadlock. Both services go to zero, each waiting for the other to recover. You need a manual fix every time.

klink solves this with CoSuspended detection:

When klink scales payments to zero because database failed, it marks payments as CoSuspended — intentionally scaled down by klink, not actually broken.

When database checks its dependencies, it sees payments at zero — but recognizes it as CoSuspended and doesn't cascade. When you manually restore database, klink automatically restores payments.

No deadlock. No manual intervention.

Argo Rollout Support — Canary Awareness

klink understands Argo Rollouts. If your payments service is in the middle of a canary deployment when its database goes down, klink defers the scale-to-zero until the rollout completes.

You never want to interrupt an active deployment. klink handles this automatically.

CronJob Support — Suspend Instead of Scale

For batch jobs, scaling to zero makes no sense. klink sets spec.suspend: true instead:

spec:
  dependent:
    kind: CronJob
    name: nightly-billing-export
  dependsOn:
    - kind: Deployment
      name: billing-service

When billing-service goes down, the CronJob is suspended. No failed jobs accumulating in history. When billing-service recovers, the CronJob resumes automatically.

Notifications

Get notified when workloads are suspended or restored — before your monitoring fires:

spec:
  notify:
    webhookSecretRef:
      name: slack-webhook
      key: url
    onPhases: [Suspended, Healthy]

The notification arrives the moment klink acts, with full context:

{
  "workloadDependency": "payments-needs-database",
  "namespace": "production",
  "phase": "Suspended",
  "previousPhase": "Degraded",
  "dependent": "payments-service",
  "dependentKind": "Deployment",
  "message": "dependency postgresql not healthy: 0/3 ready",
  "timestamp": "2026-06-15T03:00:00Z"
}

Notifications include retry with exponential backoff (1s → 2s → 4s) so transient webhook outages don't silently drop alerts.

Safety Net — maxSuspendDuration

Long outages happen. Your database might be down for hours. You don't want your payments service suspended indefinitely.

spec:
  onDegraded:
    action: ScaleToZero
    maxSuspendDuration: 4h

After 4 hours, klink restores the workload regardless of dependency state and enters Released phase — it won't re-suspend until the dependency genuinely recovers. This prevents indefinite outages from a single bad dependency.

Observability

klink exports Prometheus metrics so you can see exactly what's happening:

klink_dependency_phase{namespace="production", name="payments-needs-database", phase="Suspended"} 1
klink_scale_to_zero_total{namespace="production", kind="Deployment", name="payments-service"} 3
klink_replicas_restored_total{namespace="production", kind="Deployment", name="payments-service"} 3

GKE users get a PodMonitoring resource automatically when metrics are enabled.

Getting Started

helm upgrade --install klink oci://ghcr.io/n0rm4l-me/charts/klink \
  --version 0.3.0 \
  --namespace klink-system \
  --create-namespace

Apply your first WorkloadDependency:

apiVersion: deps.klink.dev/v1alpha1
kind: WorkloadDependency
metadata:
  name: payments-needs-database
  namespace: default
spec:
  dependent:
    kind: Deployment
    name: payments
  dependsOn:
    - kind: Deployment
      name: database
      condition:
        minReadyPercent: 80
        window: 30s
        recoveryWindow: 60s
  onDegraded:
    action: ScaleToZero
  mode: observe  # start here — see what klink would do

Check the status:

kubectl get workloaddependencies -A

NAMESPACE    NAME                      PHASE     REPLICAS   MESSAGE
production   payments-needs-database   Healthy              all dependencies healthy

When you're comfortable with what you see in observe mode, switch to strict or soft.

What klink Supports

Workload	As dependent	As dependency
Deployment	✅ scale to 0	✅ readyReplicas check
StatefulSet	✅ scale to 0	✅ readyReplicas check
CronJob	✅ suspend/resume	—
Argo Rollout	✅ scale to 0 (canary-aware)	✅ phase check

The Incident That Started This

We run microservices on Kubernetes. One evening our message queue had a rolling restart — routine maintenance, 45 seconds of unavailability. But 12 services that depended on it kept running and kept trying to connect. By the time the queue was back, we had retries queued up, connection pools exhausted, and a 10-minute degraded period that should have been 45 seconds.

The fix was conceptually simple: "if the queue is down, pause the consumers." But there was no Kubernetes-native way to express that relationship.

So we built klink.

What's Next

Prometheus-based health conditions — promQuery: 'pg_up == 1' instead of readyReplicas
kubectl klink plugin — klink graph, klink why payments-service
DaemonSet support

The project is open source under Apache 2.0. Issues, PRs, and feedback welcome.

GitHub: github.com/n0rm4l-me/klink

We replaced etcd with Google Cloud Spanner. Here's what happened.

Petr Petrenko — Tue, 09 Jun 2026 04:08:55 +0000

spanner-etcd is an open source (Apache 2.0), drop-in etcd v3 replacement backed by Google Cloud Spanner. Same API, no client changes — just point --etcd-servers at it.

We built it because etcd has a fundamental scaling constraint: every write serializes through a single global revision counter. One row, one lock, every transaction waits in line. At 32 concurrent writers, that counter becomes the bottleneck.

The GKE team solved this years ago internally to scale Kubernetes to 65,000 nodes. Their implementation is closed. So we built an open one.

This is the story of how it works, what we got wrong, and the honest benchmark numbers.

The core idea: timestamps as revisions

etcd's revision is a monotonically increasing integer. Every write increments it. That increment is the serialization point.

Spanner has PENDING_COMMIT_TIMESTAMP() — a TrueTime-based timestamp assigned at commit time, globally unique, strictly monotonic across all transactions. No counter. No lock. Each transaction commits independently.

So instead of:

UPDATE kv_rev SET rev = rev + 1 WHERE id = 1;  -- everyone waits here
INSERT INTO kv (rev, key, value) VALUES (42, '/foo', 'bar');

We do:

INSERT INTO kv (rev, key, value)
VALUES (PENDING_COMMIT_TIMESTAMP(), '/foo', 'bar');

The revision is the commit timestamp, cast to int64 UnixNano. Valid etcd ModRevision. Zero contention.

Result: at ×32 concurrency, write throughput went from a serialized bottleneck to 673 ops/sec — 15× faster than the integer counter baseline.

Watch events via Change Streams

etcd Watch is a streaming API — clients subscribe to a prefix and receive events as writes happen. In a vanilla etcd replacement you'd poll. We tried that first: it worked, but ~1s latency felt wrong.

Spanner has Change Streams — a push-based CDC mechanism that delivers row changes within tens of milliseconds. We built a partition reader that:

Starts streaming all partitions of kv_changes
Persists partition cursors to Spanner every 5s (so replicas resume correctly after restart)
Falls back to 1-second polling on the emulator (Change Streams aren't supported there)

The result: ~30ms Watch latency end-to-end on production Spanner in the same region. Not etcd's 1ms — Spanner is not a local in-memory store. But for Kubernetes workloads it's completely fine.

The back-join problem we almost missed

Early benchmarks showed Get at 71 ops/sec. Seemed reasonable. Then we looked at the query plan.

Our schema used PRIMARY KEY (id) with a bit_reversed_positive sequence — standard Spanner advice to avoid write hotspots. The secondary index kv_key_rev ON kv(key, rev DESC) existed for reads. But Spanner was doing this for every Get:

Index scan kv_key_rev → find the row's id
Table lookup on kv by id → fetch the actual data

Two round-trips inside one query. The fix was a single DDL change:

CREATE INDEX kv_key_rev ON kv (key, rev DESC)
  STORING (value, old_value, lease_id, deleted, created,
           create_revision, prev_revision);

STORING copies all needed columns into the index. Spanner can now serve reads entirely from the index — no back-join.

Get improved +40%. Mixed workload improved +167%. Measured before and after on the same hardware.

We also added kv_rev_desc ON kv(rev DESC) so CurrentRevision() does an O(1) LIMIT 1 seek instead of a full MAX(rev) scan.

One caveat: STORING value where value is BYTES(MAX) doubles write amplification for large values. For Kubernetes workloads (mostly small JSON/protobuf objects) this is fine. For blob storage it would be a problem.

Stateless replicas

This is the part that feels almost too simple. Because all state lives in Spanner, every replica is completely stateless. No consensus. No leader election between replicas. No split-brain.

We tested this explicitly: Watch on replica 2, writes through replica 1, then killed replica 1. Replica 2 received all events — before and after the kill — with zero gaps. 45 Watch streams migrated in ~10s. Kubernetes didn't notice.

The only statefulness is the Change Stream cursor, persisted to Spanner itself and recovered on restart. No leader election, no quorum, no split-brain scenario possible.

Real numbers

Everything below is production Spanner (regional-us-central1, 1000 PU), same-region e2-standard-4 VM, not the emulator.

Throughput:

Operation	ops/sec	Latency
Create ×1	90	11.1ms
Create ×4 parallel	270	3.7ms
Get ×1	108	9.3ms
Get ×4 parallel	481	2.1ms
Mixed ×4 (70% read)	403	2.5ms
Watch latency	—	~30ms

How many Spanner PUs do you actually need?

We benchmarked at 100, 1000, and 2000 PU on us-central1:

Operation	100 PU	1000 PU	2000 PU
Create ×4 parallel	87	270	255
Get ×4 parallel	472	481	469
Mixed ×4	294	403	404
Watch latency	29ms	~30ms	30ms

Interesting findings:

Single-key ops are nearly identical across 1000 and 2000 PU — you're paying for network round-trip, not Spanner compute (CPU was ~1% during benchmarks)
Parallel writes fall off sharply at 100 PU — Create ×4 drops from 270 to 87 ops/sec
Watch latency is consistent at ~30ms across all tiers
100 PU is enough for small clusters (< 100 nodes with moderate write rates)

Multi-region: what does global durability actually cost?

We ran one more experiment. We switched to nam6 — Iowa + South Carolina + Oregon + Los Angeles — and benchmarked from both regions.

The Spanner leader lives in Iowa. So writes from Iowa replicate synchronously to South Carolina before committing. Writes from South Carolina travel to Iowa, get committed, then come back.

Operation	Regional Iowa	nam6 Iowa	nam6 S.Carolina	nam6 S.Carolina + DR
Create ×1	90	53	11	11
Create ×4 parallel	270	203	45	45
Get ×1	108	116	14	16
Get ×4 parallel	481	577	60	64
Mixed ×4	403	327	49	53
Watch latency	~30ms	42ms	131ms	196ms

DR = --spanner-read-location=us-east1 — directed reads to the local South Carolina replica.

What this tells you:

Writes from Iowa get ~40% slower — that's the cost of synchronous replication to South Carolina. From South Carolina writes are 8× slower — each write travels Iowa→S.Carolina twice.

Directed reads improve read latency by 7-14% from South Carolina — reads go to the local replica instead of Iowa. The improvement is modest because writes still dominate the mixed workload, and Watch latency actually gets worse (Change Stream cursors still follow the leader path).

The practical conclusion: put your spanner-etcd replicas in the same region as the Spanner leader. If you need RPO=0 and must run replicas far from the leader, use --spanner-read-location to at least get reads locally. But writes will always pay the cross-region round-trip.

Kubernetes validation

We ran Kubernetes v1.33.12 (kubeadm, external etcd = spanner-etcd) for 24 hours straight:

Rolling deployments scaled 1–10 replicas every 2 minutes
ConfigMap churn every 3 minutes
cert-manager running concurrently
57 active Watch streams throughout

Results: zero crashes, zero data loss, zero unimplemented errors. The Kubernetes node stayed Ready the entire time.

We also tested with 22 production Java/Kotlin microservices (Vert.x + jetcd) on GKE. Auth token expiry, pod kill, Watch stream migration — all clean.

What we didn't build (and why)

Auth RBAC (UserAdd, RoleAdd, GrantPermission) — Kubernetes doesn't use it. The API server manages its own RBAC. We implement Authenticate (username/password → token) because kubeadm requires it, but the full RBAC surface isn't needed.

Defrag / Snapshot — Spanner manages storage automatically. These operations don't have a meaningful equivalent.

Sub-10ms Watch latency — if you need this, spanner-etcd is the wrong tool. Change Streams have inherent latency. For most Kubernetes operations this doesn't matter — the API server isn't latency-sensitive to etcd Watch at the millisecond level.

What surprised us

The covering index made a bigger difference than the PCT revision change. We expected the write bottleneck removal to be the headline. It was. But the read path optimization nearly doubled read throughput and nearly tripled mixed workload numbers. Sometimes the boring infrastructure work matters more than the clever architectural idea.

100 PU is genuinely enough for most clusters. We expected a linear relationship between PU and performance. Instead we found that network latency dominates and Spanner CPU is barely touched. The PU floor matters for parallel writes, but a small cluster doesn't need 1000 PU.

Non-leader regions are expensive. Iowa→South Carolina adds ~80ms round-trip. In a multi-region setup, where you place your replicas relative to the Spanner leader matters a lot more than how many PUs you provision.

Try it

docker run --rm \
  -e SPANNER_DATABASE=projects/P/instances/I/databases/D \
  -p 2379:2379 \
  ghcr.io/n0rm4l-me/spanner-etcd:v0.1.0

Or with kubeadm:

etcd:
  external:
    endpoints:
      - http://spanner-etcd:2379

GitHub: github.com/n0rm4l-me/spanner-etcd