Search for workflow automation on Kubernetes and you get one answer, repeated: run n8n in a cluster. It works, and the guides are good. But read past the install step and they all arrive at the same place — to scale past one replica you need queue mode, which means Redis, a separate worker Deployment, a separate webhook Deployment, and three environment variables set on the workers and deliberately not on the main instance.
Get that wrong and the failure is quiet: every instance activates its own copy of every trigger, and your nightly job runs once per pod.
That's not a criticism of n8n — it's the shape of the problem. A workflow engine built as a UI application has to bolt distribution on afterwards. This post is about the other approach: workflows that are already a normal Deployment, where scaling out is replicas: 3 and nothing else changes.
Time: about 15 minutes. You'll need: a Kubernetes cluster (k3s, kind and k3d all work), kubectl, and an Air Pipe API key — the free tier is enough. Postgres comes with the manifest below, so there is nothing else to provision.
Everything here was run end to end on a four-node k3d cluster (one server, three agents) against airpipeio/agent:1.42.1. The output blocks are that run, not illustrations.
The failure you're trying to avoid
Three replicas, one cron line. Each pod holds its own copy of the config, so each pod's scheduler sees that line and wants to run it. Without coordination, all three do.
Here is that happening, measured on that cluster:
minute | runs | pods
------------------------+------+------
2026-08-07 17:14:00+00 | 7 | 3 ← one cron line, seven executions
Seven, not three, because a pod that restarts re-registers. If that job sends invoices, you have just billed everyone twice and one customer three times.
Nothing is broken. Every pod is doing exactly what it was told. What is missing is somewhere for them to agree.
Give the pods one shared Postgres and one line declaring they are a cluster, and their schedulers race for a durable claim on (config, interface, minute). Exactly one wins and runs the job; the rest skip that tick. A pod that restarts re-registers and still loses the race, so seven attempts stay one execution.
That is the whole fix, and it is what the manifest below sets up.
Deploy it
One manifest: a namespace, a throwaway Postgres, the RBAC for pod discovery, a headless Service for the mesh, a config with two routes, and the Deployment at three replicas. Replace the API key and apply it.
airpipe.yaml
If any issues with the yaml, copy from -> https://airpipe.io/blog/workflow-automation-on-kubernetes
apiVersion: v1
kind: Namespace
metadata: { name: airpipe }
---
# Postgres, so this page is followable end to end. It is ephemeral and
# single-replica — fine for coordinating a cron here, not a production database.
apiVersion: apps/v1
kind: Deployment
metadata: { name: postgres, namespace: airpipe }
spec:
replicas: 1
selector: { matchLabels: { app: postgres } }
template:
metadata: { labels: { app: postgres } }
spec:
containers:
- name: postgres
image: postgres:16-alpine
env:
- { name: POSTGRES_PASSWORD, value: airpipe }
- { name: POSTGRES_USER, value: airpipe }
- { name: POSTGRES_DB, value: airpipe }
- { name: PGDATA, value: /tmp/pgdata }
ports: [{ containerPort: 5432 }]
readinessProbe:
exec: { command: ["pg_isready", "-U", "airpipe"] }
initialDelaySeconds: 5
---
apiVersion: v1
kind: Service
metadata: { name: postgres, namespace: airpipe }
spec:
selector: { app: postgres }
ports: [{ port: 5432, targetPort: 5432 }]
---
apiVersion: v1
kind: Secret
metadata: { name: airpipe, namespace: airpipe }
stringData:
database-url: "postgresql://airpipe:airpipe@postgres.airpipe.svc.cluster.local:5432/airpipe"
mesh-token: "change-me-to-a-real-secret"
---
apiVersion: v1
kind: ServiceAccount
metadata: { name: airpipe, namespace: airpipe }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: airpipe-discovery, namespace: airpipe }
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: airpipe-discovery, namespace: airpipe }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: Role, name: airpipe-discovery }
subjects:
- { kind: ServiceAccount, name: airpipe, namespace: airpipe }
---
apiVersion: v1
kind: ConfigMap
metadata: { name: airpipe-configs, namespace: airpipe }
data:
cluster.yml: |
name: ClusterDemo
global:
databases:
app:
driver: postgres
conn_string: a|env::AIRPIPE__DATABASE_URL|
interfaces:
# Health-check every replica, including this one.
cluster/health:
output: http
method: GET
actions:
- name: Peers
discover:
kubernetes:
label_selector: app.kubernetes.io/name=airpipe
port_name: http
ready_only: true
- name: CheckAll
run_when_succeeded: [Peers]
lookup: a|Peers|
lookup_partition: true
actions:
- name: Livez
http:
url: a|body::url|/livez
timeout: 3s
post_transforms:
- extract_value: body
# A cron that must fire ONCE across the whole Deployment, not once per pod.
# Every run writes a row, so you can count them.
cluster/tick:
output: http
method: GET
schedule:
cron: "* * * * *"
enabled: true
actions:
- name: Record
database: app
query: |
CREATE TABLE IF NOT EXISTS ticks (
id BIGSERIAL PRIMARY KEY,
pod TEXT NOT NULL,
minute TIMESTAMPTZ NOT NULL DEFAULT date_trunc('minute', now())
);
multi: true
- name: Insert
run_when_succeeded: [Record]
database: app
query: "INSERT INTO ticks (pod) VALUES ($1) RETURNING id, pod, minute"
params:
- a|env::HOSTNAME->default(unknown)|
---
apiVersion: v1
kind: Service
metadata: { name: airpipe-mesh, namespace: airpipe }
spec:
clusterIP: None
selector: { app.kubernetes.io/name: airpipe }
ports: [{ name: http, port: 4111, targetPort: 4111 }]
---
apiVersion: v1
kind: Service
metadata: { name: airpipe, namespace: airpipe }
spec:
selector: { app.kubernetes.io/name: airpipe }
ports: [{ name: http, port: 4111, targetPort: 4111 }]
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: airpipe, namespace: airpipe }
spec:
replicas: 3
selector: { matchLabels: { app.kubernetes.io/name: airpipe } }
template:
metadata: { labels: { app.kubernetes.io/name: airpipe } }
spec:
serviceAccountName: airpipe
containers:
- name: airpipe
image: airpipeio/agent:1.42.1
args:
- server
- --address=0.0.0.0
- --port=4111
- --config-dir=/app/configs
- --api-key=$(AIRPIPE_API_KEY)
- --log-level=warn
ports:
- { name: http, containerPort: 4111 }
env:
- name: AIRPIPE_API_KEY
value: "REPLACE-WITH-YOUR-API-KEY"
- name: AIRPIPE__CLUSTERED
value: "true"
- name: AIRPIPE__DATABASE_URL
valueFrom: { secretKeyRef: { name: airpipe, key: database-url } }
- name: AIRPIPE__WS_MESH_TOKEN
valueFrom: { secretKeyRef: { name: airpipe, key: mesh-token } }
- { name: AIRPIPE__WS_MESH_DNS, value: "airpipe-mesh" }
- { name: AIRPIPE__WS_MESH_PORT, value: "4111" }
- name: AIRPIPE__WS_MESH_SELF
valueFrom: { fieldRef: { fieldPath: status.podIP } }
- name: POD_IP
valueFrom: { fieldRef: { fieldPath: status.podIP } }
# The root filesystem is read-only, so the agent needs a writable
# place for its identity. Without this every pod start re-enrols.
- { name: AIRPIPE_STATE_DIR, value: /tmp/airpipe }
volumeMounts:
- { name: configs, mountPath: /app/configs, readOnly: true }
- { name: state, mountPath: /tmp/airpipe }
readinessProbe:
httpGet: { path: /readyz, port: 4111 }
initialDelaySeconds: 3
livenessProbe:
httpGet: { path: /livez, port: 4111 }
initialDelaySeconds: 5
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
volumes:
- name: configs
configMap: { name: airpipe-configs }
- name: state
emptyDir: {}
kubectl apply -f airpipe.yaml
kubectl -n airpipe rollout status deploy/airpipe
deployment "airpipe" successfully rolled out
Three pods, one Postgres. The rest of this post is what each part of that file is doing and why.
Expect a restart on the first apply. The agents start before Postgres is accepting connections, and a clustered node that cannot reach its coordination database exits rather than running uncoordinated — see step 1. Kubernetes restarts them, they connect, and it settles. A restart on whichever agents lost that race is this, not a fault — across runs of this manifest it has been anywhere from none of them to all three, and a pod that loses twice shows restarts=2. A real deployment adds an init container that waits for the database so the first minute does not look like an incident.
What the first minute looks like in the logs. Two lines are expected here and neither means something is broken:
-
DB connection failed at startup/cannot reach AIRPIPE__DATABASE_URL— the Postgres race above. It clears on the restart. -
mesh peer discovery failed — keeping cached peerswithpeer_count: 0— the headless Service has no endpoints until the pods are Ready, so the first lookup finds nobody. It self-corrects;cluster/healthproves it.
That is the whole list. There is no warning about state, because durable state uses that same AIRPIPE__DATABASE_URL — naming the database once covers tick coordination, run history and state together. After the first minute a healthy deployment logs nothing at all.
Check it works
kubectl -n airpipe exec deploy/airpipe -- \
wget -qO- http://127.0.0.1:4111/cluster/health
Every replica health-checks every other replica, itself included:
Peers : 3 pods
CheckAll: succeeded 3 failed 0
Then the part that matters. The config has a cron on * * * * * that writes one row per run. Leave it for a few minutes and count:
kubectl -n airpipe exec deploy/postgres -- \
psql -U airpipe -d airpipe -c \
"SELECT minute, COUNT(*) AS runs, COUNT(DISTINCT pod) AS pods
FROM ticks GROUP BY minute ORDER BY minute;"
minute | runs | pods
------------------------+------+------
2026-08-10 14:35:00+00 | 1 | 1
2026-08-10 14:36:00+00 | 1 | 1
2026-08-10 14:37:00+00 | 1 | 1
2026-08-10 14:38:00+00 | 1 | 1
2026-08-10 14:39:00+00 | 1 | 1
One run per minute across three pods. Compare that with the table at the top of this post, which is the same job with coordination switched off.
Step 1 — Say that it is a cluster
env:
- name: AIRPIPE__CLUSTERED
value: "true"
- name: AIRPIPE__DATABASE_URL
valueFrom:
secretKeyRef: { name: airpipe, key: database-url }
One declaration on every pod, meaning "this agent is one of several serving the same configs". The nodes then race for a durable claim on (config, interface, minute) in Postgres, and exactly one wins. The other two skip the tick.
Why it isn't inferred. Replica count is invisible from inside a container, and every setting that hints at clustering is legitimate on one node too — a single agent might use Postgres purely to keep run history. Guessing wrong is expensive in both directions: guess "clustered" and one node dies when an optional history store goes down; guess "standalone" and a real cluster quietly runs every job on every node. So it is a declaration, not a heuristic.
What it changes:
| Subsystem | Clustered | Not clustered |
|---|---|---|
| Scheduler | Unreachable database is fatal at startup | Warns, runs standalone |
| Realtime | Warns when cross-node fan-out is unconfigured | Silent |
| State | Requires a durable backend | Falls back to per-pod memory |
The scheduler being fatal is deliberate. It runs before the HTTP listener binds, so exiting isn't a degraded mode — the API never comes up at all, the pod restarts, backs off, and joins properly once the database answers. A node that can't coordinate can't do its job, and under an orchestrator crash-and-retry is the correct behaviour rather than a last resort.
That strictness applies only at startup. Once running, a tick-claim query that fails causes the node to skip that tick rather than run it, so a mid-life database blip can never produce a double-run. The next tick self-heals.
Step 2 — Probes that don't cost you anything
readinessProbe:
httpGet: { path: /readyz, port: 4111 }
livenessProbe:
httpGet: { path: /livez, port: 4111 }
/livez answers as soon as the process is up. /readyz answers once configs are loaded, and returns 503 with "starting" until then — which is what stops the Service sending traffic to a pod that has nothing to serve yet.
Both are dedicated routes that bypass the metered request path. Probes fire every few seconds forever; billing them as API calls would be a slow tax on running the thing correctly.
Step 3 — Configs from a ConfigMap
volumes:
- name: configs
configMap: { name: airpipe-configs }
Worth knowing what Kubernetes actually mounts here, because it bites more tools than it should: a ConfigMap volume is not a directory of files. It's a symlink farm — ..data pointing at a timestamped directory, and one symlink per key.
Walk it naively and you find every config three times, which for a scheduled workflow means registering the same cron three times within a single pod. Air Pipe skips the dot-prefixed entries, so each key loads exactly once.
A changed ConfigMap reloads in place. Edit it and re-apply; the kubelet updates the mounted files on its own sync period — usually under a minute — and the agent picks the change up without a restart:
The config lives inside airpipe.yaml above, so edit the cluster.yml block there and re-apply the file:
kubectl apply -f airpipe.yaml
If you keep your configs as separate files instead, the same thing from a standalone cluster.yml:
kubectl -n airpipe create configmap airpipe-configs \
--from-file=cluster.yml --dry-run=client -o yaml | kubectl apply -f -
Routes you removed stop serving, routes you added start — under a minute later (30 to 55 seconds across these runs), bounded by the kubelet's sync period rather than by Air Pipe. Give it that long before concluding a change did not take, including for a realtime interface you just added. Nothing restarts, so in-flight requests are undisturbed, and the restart counter stays where it was. This needs 1.41.1 or newer — before that the reload silently did nothing and a rollout was the only way.
Step 4 — Scale out
The manifest already runs three, so go further:
kubectl -n airpipe scale deploy/airpipe --replicas=5
Nothing else changes. cluster/health immediately reports five peers and five successful checks, because discovery reads the live endpoint list rather than a number you configured somewhere. The cron keeps writing exactly one row per minute — five schedulers racing for the same claim, one winner.
That's the whole scaling story. No queue mode, no Redis, no second Deployment for workers and a third for webhooks. The same pod serves HTTP routes, runs the scheduler, handles webhooks and speaks MCP, because they're all just interfaces in the same config.
Scheduled jobs coordinate through Postgres. HTTP and webhooks are stateless, so the Service load-balances them. Nothing needs pinning to a particular pod.
One thing to know if you scaled with kubectl scale and then edit your config: re-applying airpipe.yaml puts replicas back to the 3 written in it. Change the number in the file, or scale again after applying.
Step 5 — Realtime across pods
The one thing that genuinely needs wiring: a WebSocket publish arrives at whichever pod the load balancer picked, but its subscribers are spread across every pod.
- { name: AIRPIPE__WS_MESH_TOKEN, valueFrom: { secretKeyRef: { name: airpipe, key: mesh-token } } }
- { name: AIRPIPE__WS_MESH_DNS, value: "airpipe-mesh" }
- { name: AIRPIPE__WS_MESH_PORT, value: "4111" }
- { name: AIRPIPE__WS_MESH_SELF, valueFrom: { fieldRef: { fieldPath: status.podIP } } }
airpipe-mesh is a headless Service (clusterIP: None), which makes DNS return one A record per ready pod instead of a single virtual IP. The peer list refreshes in the background, so it follows scaling with no restart.
All four matter. The mesh is off when the token is unset — peers verify it, so nothing else on the network can inject events. And WS_MESH_SELF has to be exact: that DNS name resolves to this pod too, so without a self-value to exclude, a node mesh-publishes to itself and double-delivers to its own subscribers.
The manifest above wires all of this, but its config has no realtime interface to exercise it — the two routes are HTTP. To watch it work, add an interface with ws: "on", subscribe on one pod and publish on another: the message arrives, having crossed the mesh. The realtime WebSocket pack is a complete config to drop in.
Step 6 — Let a workflow see the cluster
This is the part with no equivalent elsewhere. A discover action returns the live members of a service as an array, and lookup: fans actions out across all of them:
actions:
- name: Peers
discover:
kubernetes:
label_selector: app.kubernetes.io/name=airpipe
port_name: http
ready_only: true
- name: CheckAll
lookup: a|Peers|
lookup_partition: true
actions:
- name: Livez
http: { url: a|body::url|/livez }
{ "succeeded": [ { "data": { "Livez": { "data": { "status": "ok" } } } } ], "failed": [] }
lookup_partition splits the result into succeeded and failed, so a pod that doesn't answer is reported rather than silently dropped — the difference between "all good" and "two of three replied".
Four backends return the same item shape — kubernetes, dns, docker and a static list — so a workflow written against one orchestrator moves to another by changing the backend block and nothing else.
Prefer the DNS backend where addresses are enough. It finds the same pods through the headless Service with no API access, no credentials and no RBAC. Move up to the Kubernetes backend only when you need label selectors or pod metadata, and it needs exactly one grant:
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["list"]
list on pods, nothing else. No watch, no get, nothing on secrets. It's a namespace-scoped Role, so discovering another namespace means a RoleBinding in that namespace — which keeps every grant visible where it applies.
What you get without asking
From the same config, no extra deployment: an OpenAPI document generated from the assert tests, Prometheus metrics per route, and an OpenTelemetry trace per request showing each action and how long it took. When a nightly job gets slow, the trace says whether it was your database or the API you called, before you start guessing.
The shortcut
Everything in step 6 ships as one pack — Kubernetes Service Discovery — with four routes: list pods by label with node, image and readiness, find peers by DNS with no RBAC at all, health-check every replica, and broadcast one call to the whole Deployment. Install it, apply the RoleBinding above, and the routes work against your own cluster.
Not on Kubernetes? The same shape over Docker, Compose or plain DNS is Container Service Discovery. Every option each backend takes is in the discovery reference.
Things that will bite you
sessionAffinity: ClientIP will pin you to one pod. It's a reasonable-looking default that quietly turns three replicas into one — a load test looked beautifully stable until we noticed 12 of 12 requests hit the same pod. Leave it None unless you specifically need stickiness.
NetworkPolicy and NodePort. A default-deny policy with only a namespaceSelector will drop external traffic, because NodePort traffic arrives SNAT'd from the node rather than from a pod in a namespace you matched. If your Service works from inside the cluster and not from outside, that's usually why. And k3s does enforce NetworkPolicy — it ships kube-router for exactly that.
A read-only root filesystem needs somewhere to write. Self-hosted agents sign their usage reports with a per-agent keypair kept on disk. With readOnlyRootFilesystem: true the enrolment succeeds and only the persist fails, so nothing looks broken while every pod start mints a fresh identity. Give it AIRPIPE_STATE_DIR=/tmp/airpipe on an emptyDir, or supply a pre-enrolled identity.
One agent, one node, still needs the declaration. AIRPIPE__CLUSTERED is a product-level flag, not a scheduler one — realtime and durable state read it too. Setting it on a single-replica Deployment is harmless and makes the intent explicit if you ever scale up.
Where this fits
If your automation is a person clicking nodes together, a visual builder is worth its operational cost, and n8n is very good at that.
If your automation is closer to "a scheduled job, a webhook and an internal API that share the same database" — the things that end up in a workflow tool because there was nowhere else to put them — then it can be one config file, in the same Git repo as everything else, deployed as a normal Deployment and scaled with kubectl scale.
The manifest above is deliberately one file so it can be read in one sitting. For a real deployment — kustomize overlays, a Helm chart, resource limits, an Ingress and the managed-mode variant — see the Kubernetes install guide.
Originally published at airpipe.io.
Top comments (0)