No cloud to lean on: building a reliable data platform on a bank's own servers

Replication, failover, graceful shutdown, autoscaling and alerting usually arrive as managed services. Inside a bank, you build them from Kubernetes, an operator or two and a lot of small decisions. Here is how each one works, with the code.

Field notes from moving a transaction reconciliation platform from AWS onto Kubernetes inside banks' data centres

1. The constraints
2. Five words, one goal
3. Reliability: no work is lost silently
4. Fault tolerance: replication, redundancy, failover, graceful degradation
5. High availability: what survives what
6. Scalability on a fixed number of servers
7. Monitoring and observability without the internet
8. High availability is not a backup
9. Checklist

The constraints

In the cloud, our reconciliation platform ran on managed pieces: a managed database with automatic failover, managed queues, managed object storage, autoscaling groups and a monitoring SaaS. Banks wanted the same platform on their own hardware and their hardware comes with rules:

• No internet. Images come from the bank's registry, backups go to the bank's storage and the only way out is the bank's mail relay.
• A fixed number of servers. Three to seven machines, sized months in advance. Nothing appears when load goes up.
• No managed services. Every database, queue and object store runs inside the cluster, so failover is our problem.
• Little access. The people on call may not have cluster access and our engineers get in only through a bastion, when the bank allows it.

The platform itself is a typical event-driven data system. Users upload files through a web portal. Workers clean them, an ingestion engine loads them into an Iceberg data lake with Spark and reconciliations match transactions across sources. Every hop between services goes through a message queue.

Figure 1. The platform inside one Kubernetes cluster. The only things that leave are backups, email and image pulls and each goes to a system the bank controls.

Five words, one goal

These terms get used loosely, so here is what each one meant for us. They overlap and they all serve one goal we wrote at the top of the design: losing any one machine loses no data and stops nothing for more than seconds.

Concept	The question it answers	What it looks like here
Reliability	Does the system do the right thing, every time?	Work is acknowledged only after it is done; locks can't be taken twice; shutdown never drops a file
Fault tolerance	What happens when a part fails?	Replicated database and queues, spread replicas, automatic failover, failures contained to one job
High availability	How long are users affected?	Seconds for a lost server; zero for upgrades and node maintenance
Scalability	Can it keep up when the work grows?	Replica counts follow queue depth; heavy jobs wait for room instead of failing
Observability	Will someone find out and can they tell why?	Alerts by email, searchable logs, dashboards, a daily health report

Reliability: no work is lost silently

The rule underneath everything else: work moves through the queue and a message leaves the queue only when its work is done. A service that is down or busy leaves messages waiting. Nothing is lost while it recovers.

Acknowledge after the work, never before

A message broker like RabbitMQ hands a message to one consumer and marks it unacknowledged. It stays in the queue until the consumer says what happened. That gives you three outcomes and each one needs a deliberate answer.

Figure 2. Every path out of "unacknowledged" is deliberate. The message disappears only on success; everything else either retries or lands somewhere a person will see it.

Here is the pattern in a small consumer that forwards events from a queue to an HTTP endpoint. It acknowledges only after the endpoint answers 2xx and on any failure it puts the message back for another attempt:

Python (kombu)

def forwarder(endpoint, http_timeout):
    def on_message(body, message):
        try:
            response = requests.post(endpoint, data=to_payload(body), timeout=http_timeout)
            response.raise_for_status()
            message.ack()                      # only now does it leave the queue
        except Exception:
            logger.exception("Failed to forward message; requeueing")
            message.reject(requeue=True)       # back to Ready, tried again
    return on_message

Two settings make that safe at the broker. Quorum queues in RabbitMQ drop a message after 20 failed deliveries by default. A consumer that requeues on purpose while its downstream is down (the portal restarting, say) would lose messages after a few minutes of retries, so we lift the limit with a policy and rely on dead-letter queues and alerts for work that truly can't succeed:

curl -X PUT "$api/policies/%2F/redelivery" -H 'content-type: application/json' \
  -d '{"pattern": ".*", "apply-to": "quorum_queues",
       "definition": {"delivery-limit": -1}, "priority": 0}'

Watch out for the poison message. Unlimited redelivery has a dark side. A message whose processing kills the consumer is redelivered to the next consumer, which dies too.

Figure 3. The poison loop we hit on a test server. One large file was redelivered from pod to pod, killing each one at its memory limit.

The fix was sizing, not code: the file cleaner's memory limit went from 2 GiB to 4 GiB and a day later the pods still had no restarts. The lessons are general, though:

• Size memory limits for the largest input, not the average one. An OOM kill is instant: no exception handler runs, nothing is logged by the app and the message goes back to the queue.
• Alert on OOM kills and on restarts, so a loop shows up in minutes rather than as "the system is slow today".
• Stream large inputs where you can, so memory stays flat whatever the file size.

Consumers that can't stall silently

A web service has an obvious health check: send it a request. A queue consumer has nothing to probe and its worst failure leaves the process running. The connection drops, the consume loop blocks forever, the pod shows Running and the queue grows behind it.

Figure 4. A heartbeat written only by the loop that does the work turns "running but useless" into a restart.

The consumer touches a file at the end of each pass through its loop. Long jobs run on their own threads, so a big file never holds the loop up and the heartbeat really does mean "this process is still taking work":

Python

LIVENESS_FILE = pathlib.Path("/tmp/heartbeat")

while not stopping:
    try:
        conn.drain_events(timeout=5)
    except socket.timeout:
        pass
    conn.heartbeat_check()
    LIVENESS_FILE.touch()          # only a working loop gets here

Kubernetes checks the file's age. A startup probe gives the process time for its first beat; after that, three stale checks in a row restart it:

Kubernetes

startupProbe:
  exec:
    command: ["sh", "-c", 'f=/tmp/heartbeat; [ -f "$f" ] && [ $(( $(date +%s) - $(stat -c %Y "$f") )) -lt 120 ]']
  periodSeconds: 10
  failureThreshold: 30          # up to 5 minutes for the first heartbeat
livenessProbe:
  exec:
    command: ["sh", "-c", 'f=/tmp/heartbeat; [ -f "$f" ] && [ $(( $(date +%s) - $(stat -c %Y "$f") )) -lt 120 ]']
  periodSeconds: 30
  failureThreshold: 3

Probes can cause outages too. RabbitMQ's usual readiness check, rabbitmq-diagnostics ping, starts a whole separate Erlang runtime to answer. On a busy node that timed out while the broker was healthy, which pulled it out of the Service and refused every client. A plain tcpSocket check is a kernel-level connect with no process to start and it stays reliable under the same load.

Locks that can't be taken twice

Two reconciliations on the same account at the same time would write conflicting results, so each run locks its accounts first. The obvious way ("check whether a lock exists, then write one") lets two triggers that arrive together both see "unlocked". The lock has to be a single conditional write and it has to record who owns it:

SQL (illustrative)

-- Take the lock only if nobody holds a live one. Exactly one caller gets a row back.
INSERT INTO account_locks (account_id, owner, expires_at)
VALUES (:account, :owner, now() + interval '1 hour')
ON CONFLICT (account_id) DO UPDATE
  SET owner = EXCLUDED.owner, expires_at = EXCLUDED.expires_at
  WHERE account_locks.expires_at < now()          -- take over only a stale lock
RETURNING account_id;

-- Release only your own lock, never someone else's.
DELETE FROM account_locks WHERE account_id = :account AND owner = :owner;

• Lock several accounts in a fixed order and on a conflict release the ones already taken. Two runs grabbing the same accounts in different orders is the classic deadlock.
• Every lock-holding workflow releases its locks on failure, on every step. Otherwise a failed run blocks the account until the lock expires.
• Keep the lock table in the replicated, backed-up database. We moved this state from DynamoDB in the cloud to a Postgres table behind the same interface. Adding a separate store would have meant one more thing to make highly available and back up.

Things that must happen once

Some work must never run twice at the same moment and Kubernetes happily starts several copies of everything.

Database migrations. On a first install, or when scaling up from zero, every portal pod starts together and concurrent migrate runs on an empty database fail on duplicate keys. Each pod's init container takes a Postgres advisory lock first. One pod migrates and the others wait and then find nothing left to do:

Python (init container)

import django
django.setup()
from django.core.management import call_command
from django.db import connection

with connection.cursor() as cursor:
    # Held by this session until the process exits.
    cursor.execute("SELECT pg_advisory_lock(hashtext('portal-migrate'))")
call_command("migrate", interactive=False)

The scheduler. Two copies of a periodic-task scheduler fire every task twice. It runs as one replica with the Recreate strategy, so even an upgrade stops the old pod before starting the new one. Everything else rolls with a surge, so a new pod is ready before an old one goes:

# scheduler: never two at once
strategy:
  type: Recreate

# portal: never fewer than the desired count during a rollout
strategy:
  type: RollingUpdate
  rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }

Graceful shutdown: finish the work in hand

Pods are stopped all the time: every upgrade, node patch and scale-down. Kubernetes sends SIGTERM, waits for the grace period, then kills the process. A worker that ignores SIGTERM loses whatever it was doing on every deployment.

Figure 5. Graceful shutdown has two good endings. Either the work finishes and is acknowledged, or it is left unacknowledged and redone elsewhere. The bad ending, a half-finished file marked as done, can't happen.

Python

stopping = False

def on_sigterm(signum, frame):
    global stopping
    stopping = True                  # finish the current message, then leave

signal.signal(signal.SIGTERM, on_sigterm)

with Connection(broker_url, heartbeat=60) as conn, conn.Consumer(queues, callbacks=[on_message]):
    while not stopping:
        try:
            conn.drain_events(timeout=5)   # returns at least every 5s to check the flag
        except socket.timeout:
            pass

Two details are easy to miss:

• exec the real process in a shell entrypoint (exec celery worker ...). Otherwise SIGTERM goes to the shell, which ignores it and the worker is killed at the end of the grace period with no warning.
• Know when your framework acknowledges. Celery acknowledges a task when it starts by default, so a task cut off by SIGKILL is gone. acks_late fixes that, but then a task runs again after a crash, which is only safe when every task can run twice. We left it off until every task is idempotent.

Component	On SIGTERM	Grace period
File and report workers	Stop consuming, finish and acknowledge the job in hand	900s / 1,800s
Celery workers	Stop taking tasks, finish running ones	900s
Ingestion engine	A preStop drain: stop picking up runs, wait for its own to finish	1,800s
Event forwarder, notifier	Finish the current message or email	60s
Message broker node	Wait until every queue keeps a majority without it (see failover below)	Up to the grace period
SQL engine worker	Finish running query tasks	Until its tasks end

Fault tolerance

Fault tolerance is four techniques working together. Replication keeps the data in more than one place. Redundancy keeps more than one copy of each service running, on different machines. Failover moves the work to a healthy copy when one dies. Graceful degradation keeps the rest working, more slowly if it must, when something can't be replaced.

Replication: the database

Postgres runs as a three-instance cluster managed by the CloudNativePG operator: one primary and two streaming replicas, each on a different server. Every service connects to one Service name (postgres-rw) and the operator always points it at the current primary.

We chose it because it uses the Kubernetes API itself to decide which instance is primary. There is no extra consensus store (Patroni-based setups need etcd or similar) and no failover scripts of our own. WAL archiving and point-in-time recovery are built in, it's a CNCF project and it's on OpenShift's OperatorHub, which matters to banks that run OpenShift.

Figure 6. Synchronous, quorum-style replication. A commit the application was told about survives the loss of any one server.

CloudNativePG Cluster

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: postgres
spec:
  instances: 3
  primaryUpdateStrategy: unsupervised   # roll replicas, then switch over, no human needed
  smartShutdownTimeout: 15              # default 180s: see below
  enablePDB: true
  affinity:
    enablePodAntiAffinity: true         # one instance per server
  postgresql:
    synchronous:
      method: any                       # synchronous_standby_names = 'ANY 1 (...)'
      number: 1
      dataDurability: preferred         # degrade to async rather than stop writes

preferred rather than required is a graceful-degradation choice. With required, every write stops while no replica is up and a rolling update or a lost server would stop the platform. The window where only one copy exists is short and an alert says when it opens.

One default worth changing: smartShutdownTimeout. A stopping primary waits for open connections to close before a replica takes over. Django keeps its database connections open, so the default 180 seconds meant three minutes of errors on every primary restart. Fifteen seconds is plenty.

Replication: the message queues

RabbitMQ runs as a three-node cluster with quorum queues: each queue is a Raft group copied on all three nodes. A message is confirmed to the publisher once a majority has it, so the queue keeps working and loses no confirmed message, while any two nodes are up.

Figure 7. A quorum queue tolerates one lost node out of three. Two lost nodes stop it, by design, rather than risk two diverging copies.

rabbitmq.conf

# Every node finds the others through the Kubernetes API.
cluster_formation.peer_discovery_backend = k8s
cluster_formation.k8s.address_type = hostname

# Every durable queue a client declares without a type becomes a quorum queue,
# so no client code has to change.
default_queue_type = quorum

# A queue declared while the cluster was still forming gains replicas as nodes join.
quorum_queue.continuous_membership_reconciliation.enabled = true
quorum_queue.continuous_membership_reconciliation.target_group_size = 3

Setting the default at the broker was the important part. Every service that declares queues kept its code unchanged. Celery was the one exception. Without declaring quorum queues itself, it asks for a global QoS setting that quorum queues reject and it can't consume at all. One setting fixed it: CELERY_QUORUM_QUEUES, read from the environment.

Redundancy: copies on different machines

Three copies of a service on one server are one failure away from zero. Every workload with more than one replica prefers to spread across servers and has a PodDisruptionBudget, so a node drain (patching, a hardware swap) can't take every copy at once.

Figure 8. Spreading puts each copy on a different server. The disruption budget makes a drain wait whenever taking the next pod would leave too few copies.

Both come from two small Helm helpers that every Deployment includes:

Helm helpers

{{- define "chart.spreadReplicas" -}}
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:   # prefer, so a 1-server install still schedules
      - weight: 100
        podAffinityTerm:
          topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels:
              app: {{ .app }}
{{- end -}}

{{- define "chart.pdb" -}}
{{- if gt (int .replicas) 1 }}   # a PDB over one replica would block drains forever
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: {{ .app }}
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: {{ .app }}
{{- end }}
{{- end -}}

Both helpers handle the small case on purpose. Some installs are a single server for user acceptance testing. A hard anti-affinity rule would leave pods Pending there and a disruption budget over one replica would make every drain hang.

Failover: when a copy dies

The database fails over by itself. When the primary's pod or server is lost, CloudNativePG promotes the most up-to-date replica within seconds and moves postgres-rw to it. Clients see their connections drop and reconnect. With synchronous replication, no committed transaction is lost. The old primary rejoins as a replica when it comes back.

The broker needs care at planned shutdowns. Stopping a RabbitMQ node while another is already down would take queues below a majority. So before a node stops, a preStop hook waits until every quorum queue would still have a majority without it, then moves its queue leaders away:

StatefulSet container

terminationGracePeriodSeconds: 600
lifecycle:
  preStop:
    exec:
      # Wait for quorum without this node, then hand off its leaders.
      # Gives up before the grace period ends, so a node can still stop
      # if another one is down for good.
      command: ["/bin/sh", "-c",
        "rabbitmq-upgrade await_online_quorum_plus_one -t 540; rabbitmq-upgrade drain -t 30 || true"]

Stateless workers with long-running state are the hard case. The ingestion engine runs two or more pods, each running reconciliation workflows that hold account locks. If a pod dies mid-run, someone has to notice, mark its runs as failed and free its locks. Nobody else can tell a dead pod from a busy one, so each pod writes a heartbeat row and every pod checks the others':

Figure 9. Peer failure detection with heartbeats. Any surviving pod can clean up after a dead one and it frees exactly the locks the dead pod's runs held.

The subtle part is not judging others while you're the one in trouble. If the database is down for two minutes, every pod's heartbeat fails and when it returns every heartbeat looks expired. A pod therefore judges others only after its own heartbeats have succeeded for a full timeout period. By then every live pod has written a fresh row too:

Python

class RunnerHeartbeat:
    def __init__(self, table, runner_id, clock=time.monotonic):
        self.table, self.runner_id, self.clock = table, runner_id, clock
        self.healthy_since = clock()

    def beat(self):
        try:
            self.table.put(runner_id=self.runner_id, expires_at=int(time.time()) + HEARTBEAT_TTL)
        except Exception:
            self.healthy_since = self.clock()      # our own view is unreliable: start over
            logger.exception("Runner heartbeat write failed")

    def may_reap(self):
        """True once our own heartbeats have succeeded for a full TTL:
        any live runner has then written a fresh row too."""
        return self.clock() - self.healthy_since >= HEARTBEAT_TTL

Crash and alert, don't re-run. A dead pod's runs are marked Crashed, not retried automatically. A reconciliation stopped half-way may have written part of its results and running it again isn't known to be safe. Nothing is lost silently: the run shows as Crashed, the account is unlocked and a person is told. Automatic retry is a feature you earn by making the work idempotent.

Graceful degradation: fail small, slow down rather than fall over

Some failures can't be hidden. The aim then is that they stay small and that overload produces waiting rather than errors.

Figure 10. Isolation as graceful degradation. Running each heavy job in its own pod keeps a failure the size of the job.

• Heavy jobs fail alone. We considered Spark standalone (driver inside the service) and Spark's own Kubernetes mode, whose spark-submit exits 0 even when the job fails. A plain Kubernetes Job already gives a status, a deadline, retries, clean-up and logs.
• Too much work waits instead of failing. Without a limit, a burst of uploads starts every Spark Job at once. Those the cluster has no room for sit Pending with their deadline running and time out without having run. A queue in front of them (next section) turns a burst into a backlog.
• Downstream outages become retries. When the portal restarts, the event forwarder's requests fail and the messages are requeued until it's back. Users see their notifications a minute late, not never.
• Replication degrades to async rather than stopping writes (Figure 6).
• Telemetry never blocks the app. Services send traces and metrics to a gateway asynchronously. If the gateway or log store is down, the services carry on and the gateway buffers what it can.
• CPU limits slow a pod down; memory limits kill it. That asymmetry decides how to set each one, as the scalability section explains.

High availability: what survives what

High availability is the sum of the above, measured from the user's side. We kept one table in the architecture docs that answers "what happens if X fails?" and every design change had to keep it true:

If this fails	What happens	Users notice
A database pod or its server	A replica is promoted; no committed transaction is lost; services reconnect	A few seconds of errors
A message broker node	Every queue keeps a majority on the other two; no confirmed message is lost	Nothing
A portal pod or its server	The other replicas keep serving; the pod is rescheduled	Nothing
A workflow pod mid-run	Its runs are marked Crashed within 3 minutes, their locks freed, an alert sent	That run shows as Crashed
A pod being stopped (upgrade, drain)	It finishes its current work first	Nothing
A heavy job	Only that job fails, with its error in the run	One failed upload
The whole cluster	Reinstall and restore from off-cluster backups	An outage, within the agreed recovery time
The object store's disk	Not covered by HA yet: restore from the last bucket copy	An outage

That last row matters. A useful HA table lists what isn't covered, so the bank can accept it in writing or fund the fix. Ours is the object store, which still sits on a single disk. It's on the open-items list the bank signs at go-live.

Upgrades are the most common "failure" of all, so they get the same treatment. Setup work runs as ordinary Jobs named after the Helm revision, not as Helm hooks. That way helm upgrade --install --wait --atomic works end to end and a failed upgrade rolls itself back:

helm upgrade --install platform . -f values-bank.yaml -n platform --wait --atomic --timeout 20m

Scalability on a fixed number of servers

In the cloud, scaling means adding machines. A bank's cluster has the servers it has. Autoscaling there doesn't add capacity. It moves the capacity you have to whichever queue is backed up, which pays off when work arrives in bursts, as uploads and exports do.

Scale on the work waiting, not on CPU

CPU is the wrong signal for a queue consumer. A worker waiting on the database or the object store uses little CPU while messages pile up behind it.

Figure 11. Queue depth and CPU for the same worker during an upload burst. Only one of them says "more workers needed".

KEDA scales each worker Deployment on its queue length (and for one worker on CPU too, whichever asks for more). It manages the Horizontal Pod Autoscaler itself and it reads queue sizes with a broker user that can see queues but not use them:

KEDA ScaledObject

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: file-cleaner
spec:
  scaleTargetRef:
    name: file-cleaner
  minReplicaCount: 3
  maxReplicaCount: 8
  advanced:
    restoreToOriginalReplicaCount: true
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown: { stabilizationWindowSeconds: 300 }   # 5 quiet minutes before removing pods
  triggers:
    - type: rabbitmq
      metadata:
        protocol: http
        host: http://rabbitmq:15672/%2F
        queueName: file-cleaning
        mode: QueueLength
        value: "5"                                         # one pod per 5 messages waiting
      authenticationRef: { name: keda-rabbitmq-readonly }

The workflow engine is different. Its trigger queues empty almost at once into orchestrator runs, so queue length says nothing about its load. It scales on runs waiting or running, through a Postgres role that can do exactly one thing, call a function returning that count:

triggers:
  - type: postgresql
    metadata:
      userName: keda                     # EXECUTE on one function, nothing else
      dbName: orchestrator
      query: "SELECT keda_active_flow_runs()"
      targetQueryValue: "6"              # one pod per maxConcurrentRuns

One detail for reading the numbers: the autoscaler ignores differences under 10%. With 2 pods of 6 runs each, 13 runs (6.5 per pod) doesn't add a third pod; 14 does.

Admission control for heavy jobs

Scaling the workflow engine up means more Spark Jobs and those need real CPU and memory. Without a limit, more workflow pods would add load, not capacity. So the Spark Jobs go through a Kueue queue that holds a fixed budget of CPU and memory. A Job is created suspended and starts when the running Jobs leave room for it. Its deadline starts only then, so waiting no longer counts as failing:

Kueue ClusterQueue

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: spark
spec:
  queueingStrategy: BestEffortFIFO    # in order, but a small job may pass a large one that doesn't fit yet
  resourceGroups:
    - coveredResources: ["cpu", "memory"]
      flavors:
        - name: spark
          resources:
            - name: cpu
              nominalQuota: "8"       # the room kept for Spark beside everything else
            - name: memory
              nominalQuota: 24Gi

Kueue's admission webhook refuses to create any Job in the namespace while Kueue is down. That covers setup Jobs and backup runs as well as Spark Jobs. A new dependency is also a new failure mode, so Kueue runs two replicas with a disruption budget, only manages namespaces labelled for it and has its own alert after two minutes down.

Requests, limits and the throttling trap

Every container has CPU and memory requests (what the scheduler reserves) and limits (the ceiling). The two kinds of limit behave very differently:

	CPU limit	Memory limit
When exceeded	The pod is throttled: it waits for the next 100 ms period	The container is killed (OOMKilled)
What users see	Slowness	Restarts, redelivered work (Figure 3)
How we set it	Enough for the process's real parallelism	The largest input plus headroom; alert at 90%

The parallelism point caught us. Celery's prefork pool runs four child processes and the workers had a CPU limit of half a core. Four processes shared 500m, so every task ran at an eighth of a core. The Grafana panel "held back by CPU limit" showed it plainly. A 2-core limit matched the pool and the requests stayed small so the scheduler still packs pods tightly.

Throttling alerts have their own trap. Throttling is measured as throttled periods ÷ periods. A pod that is idle almost all the time but spikes briefly, a health check that imports a large app, say, can show 60% throttling while using 2% of its limit. Alerting on that ratio alone pages people about pods that are fine. Our alert fires only when the pod is also using a real share of its limit:

PromQL (alert rule)

(
  sum by (namespace, pod) (rate(container_cpu_cfs_throttled_periods_total{namespace="platform"}[5m]))
  / sum by (namespace, pod) (rate(container_cpu_cfs_periods_total{namespace="platform"}[5m]))
) > 0.25
and on (namespace, pod)
(
  sum by (namespace, pod) (rate(container_cpu_usage_seconds_total{namespace="platform",container!=""}[5m]))
  / sum by (namespace, pod) (max by (namespace, pod, container) (kube_pod_container_resource_limits{namespace="platform",resource="cpu"}))
) > 0.5

Monitoring and observability without the internet

Everything above assumes someone notices when something goes wrong. In a bank there's no internet, often no Slack and the people on call may have no cluster access. Our monitoring had to work entirely inside the cluster and reach people only through the bank's mail relay.

Figure 12. Three signals, one place for alerts. Logs are collected from every pod, instrumented or not; traces and metrics pass through one redacting gateway; every alert is evaluated in Prometheus and leaves by email.

Collect from everything, redact in one place

• Logs come from the container log files, not from the apps. The database, the broker and third-party components never send telemetry and they're often the ones that fail. A Vector agent on every node tags each line with its namespace, app, pod and parsed log level.
• Traces and metrics go through one gateway (the OpenTelemetry Collector). It strips what a bank must not store, such as SQL text, credentials, cookies and URL query strings, once for every service. It also buffers through a log-store restart.
• Each service is named by its app label everywhere, so its logs, traces, metrics and alerts join on one name.
• The log store keeps its data in the platform's own object store and Postgres. There's no extra disk to provision or back up.

Alerts that mean something

Prometheus evaluates the rules and Alertmanager sends email through the bank's relay. A bank's cluster can't reach Slack or PagerDuty, but it always has a mail relay. Each alert links to a runbook: what it means, what's affected, how to check, how to fix, when to escalate. A few of the rules show the thinking.

A queue with messages and nobody consuming it. This catches a dead consumer within minutes, whatever the reason:

- alert: QueueNoConsumers
  expr: |
    max by (namespace, vhost, queue) (rabbitmq_detailed_queue_messages_ready{queue!~".*(retry|dead-letter).*"}) > 0
    and on (namespace, vhost, queue)
    max by (namespace, vhost, queue) (rabbitmq_detailed_queue_consumers) == 0
  for: 10m
  labels: { severity: critical }
  annotations:
    summary: "Queue {{ $labels.queue }} has messages but no consumer for 10m"
    runbook_url: "https://runbooks.example/#queuenoconsumers"

A backup that silently stopped. This one alerts on the absence of success, not on a failure. A CronJob that never runs (suspended, a refused admission, a broken schedule) never fails either:

- alert: BackupMissing
  expr: |
    time() - (
      max by (namespace, cronjob) (kube_cronjob_status_last_successful_time{cronjob=~".*-backup-.*"})
      or max by (namespace, cronjob) (kube_cronjob_created{cronjob=~".*-backup-.*"})   # a new CronJob counts from its creation
    ) > 26 * 3600
  for: 30m
  labels: { severity: critical }

Duplicates held back. kube-prometheus-stack ships generic rules for the same failures. While our more specific alert fires for an object, the generic one for it is inhibited, so on-call gets one email per problem:

inhibit_rules:
  - source_matchers: [severity="critical"]
    target_matchers: [severity="warning"]
    equal: [alertname, namespace]
  - source_matchers: [alertname="PlatformPodCrashLooping"]
    target_matchers: [alertname="KubePodCrashLooping"]
    equal: [namespace, pod, container]

Other choices that keep the noise down:

• Error-line alerts come from a metric, not a log query. Vector counts ERROR lines per app and Prometheus scrapes the count, so log alerts share the routing, grouping and silencing of every other alert.
• Retry and dead-letter queues are left out of backlog alerts. They hold messages by design. Dead letters get their own alert.
• A Job alerts only after it has used up its retries. A setup Job that succeeds on its second try stays quiet.

Check from the users' side

Every pod can be healthy while nobody can log in: a DNS change, a load balancer, an expired certificate. A blackbox exporter fetches the portal, the object store and the other endpoints, both inside the cluster and at their public addresses. It records whether each answered, how fast and when its certificate expires. Vector also turns the portal's access log into request counts and response times, which works behind any ingress controller or OpenShift's router.

Mark every change on every chart

The first question in an incident is "what changed?". Grafana marks every Helm install, upgrade and rollback and every database failover, on every chart. They come from metrics Prometheus already has (kube-state-metrics exposes the Helm release Secrets and CloudNativePG's exporter exposes the primary), so nothing has to be recorded by hand.

A daily report, which is also a dead man's switch

Nothing reports to us from a bank on its own. So a CronJob builds a support bundle inside the cluster every morning, with read-only access. That covers pod states, events, recent logs, memory and CPU pressure, queue depths, database activity and Helm values with every credential redacted. It keeps the last 14 bundles in a bucket and emails a short report: each component's state, the problems found and what to do about each. It contains no logs and no data.

The report has a second use. When an install's report doesn't arrive by mid-morning, we ask. Its monitoring, its relay or the platform itself may be down and a platform that is down can't send alerts. A signal that must keep arriving is the only way to detect the failure of the alerting path itself.

Case study: the disk alert that wasn't about volumes. Volume-filling alerts fired on a single-server install, yet the volumes held little. On k3s, local-path volumes have no quota, so they report the node's whole disk and the disk was 82% full of old container images from months of upgrades. Pruning brought it to 58%. The lasting fix wasn't a cron job, which banks dislike on their servers. It was the kubelet's own image garbage collection, set to start at 75% disk use and stop at 65%.

# /etc/rancher/k3s/config.yaml (OpenShift: a KubeletConfig with imageGCHighThresholdPercent)
kubelet-arg:
  - "image-gc-high-threshold=75"
  - "image-gc-low-threshold=65"

High availability is not a backup

Replication protects against a machine dying. It does nothing against a bad upgrade, a deleted table or a corrupted volume: those are copied faithfully to every replica within milliseconds. That needs copies stored somewhere else and a restore that is scripted.

Figure 13. Replication and backups solve different problems. Point-in-time recovery always builds a new cluster, so the archive it reads from is never overwritten and a recovery that goes wrong can be repeated.

Three copies leave the cluster, all to the bank's own S3-compatible storage. That might be a storage appliance or a separate MinIO, ideally in another data centre. It is never the in-cluster object store, which is one of the things being backed up.

What	How	Kept
Postgres WAL and base backups	CloudNativePG's Barman Cloud plugin: each WAL segment when full or after 5 minutes; a base backup daily	35 days
Every database and role	A nightly pg_dumpall CronJob: a plain, portable copy for the scripted restore	35 days
Every bucket	A nightly or hourly rclone copy, which never deletes on the target	Never deleted

apiVersion: batch/v1
kind: CronJob
metadata:
  name: backup-postgres
spec:
  schedule: "0 1 * * *"
  concurrencyPolicy: Forbid          # a slow night never overlaps the next one
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          containers:
            - name: upload
              command: ["/bin/sh", "-c"]
              args:
                - |
                  set -e
                  rclone copy /dump target:$BUCKET/postgres
                  rclone delete --min-age 35d target:$BUCKET/postgres

Two stores must agree

An Iceberg table has two halves: data files in the object store and a catalog in Postgres that says which files make up each table. Postgres's WAL can be hours newer than the nightly bucket copy. Restored to its latest point, the catalog would point at data files the restored store doesn't have.

So the bucket copy writes a LAST_BACKUP marker with its start time to the backup target. Recovery puts Postgres back to that moment, not later. Keeping the marker off-cluster means the consistent point is still known after the whole cluster is gone. For a smaller gap, copy the buckets more often than you dump the database.

A restore you haven't run is a hope, not a plan. The restore is a script that scales everything down, refuses to continue while anything else is connected to the database, restores and scales back up. The go-live checklist includes a timed restore drill, so the recovery time the bank signs off on is a measured number.

Checklist

• Every hop between services goes through a durable, replicated queue; messages are acknowledged after the work
• Memory limits are sized for the largest input and OOM kills and restarts alert
• Queue consumers have a heartbeat written by the working loop and a liveness probe that checks it
• Locks are single conditional writes with an owner, released on every failure path
• Migrations take a lock; the scheduler runs exactly once (Recreate)
• Every process handles SIGTERM: stop taking work, finish, acknowledge, exit; grace periods fit the longest job
• The database replicates synchronously to at least one replica, on another server
• Queues are quorum queues; broker nodes wait for quorum before stopping
• Replicas spread across servers and have disruption budgets, without breaking single-server installs
• Heavy jobs run in their own pods, admitted when there is room
• Workers scale on work waiting, not CPU; CPU limits match the process's real parallelism
• Alerts arrive by a channel the site allows, each with a runbook; duplicates are inhibited
• Something must keep arriving every day, so a silent install is noticed
• Backups leave the cluster, the database and object store restore to the same moment and the restore has been timed
• The "what survives what" table is current, including what isn't covered

None of these ideas are new. What changes on a bank's servers is that no provider applies them for you, so each one becomes a line in a chart, a probe or a runbook. Writing them down, with the reason next to each, is what lets the next engineer change one without breaking the others.