Kestra for Data Engineering: YAML-First Orchestration, Plugins & UI-Driven Workflows

kestra is the pick-one architectural decision that finally puts YAML back at the centre of a modern data-engineering orchestrator — the declarative, plugin-driven, UI-first workflow engine that senior data engineers now evaluate whenever they outgrow a Python-only scheduler and cannot commit to a Kubernetes-only DSL. Every batch load, streaming enrichment, dbt run, and Airbyte sync your platform performs — a nightly Snowflake refresh, a webhook-driven feature ingestion, an event-triggered dbt build, a container-based Spark submission — has to be schedulable, observable, versionable, and safely re-runnable from any team member's editor without hard-coding a Python dependency on a single control-plane language, and it has to do all of that while running Python, SQL, JavaScript, shell, and container tasks side by side in the same flow file. The engineering trade-off does not live in "should we adopt Kestra" — every polyglot data platform needs a language-agnostic orchestrator eventually — but in which yaml orchestration engine you pick and how it composes with your existing warehouse, lakehouse, and streaming stack.

This guide is the senior-DE walkthrough you wished existed the first time an interviewer asked "walk me through kestra vs airflow and when Kestra actually wins", or "your team runs Python and SQL and JavaScript — how does one orchestrator schedule all three?", or "explain the kestra plugin model and why plugins are drop-in JARs rather than Python packages." It walks through the Kestra architecture — the YAML flow shape (id / namespace / tasks), the polyglot task catalog (Script, Sql, Http, EachParallel, Switch), Pebble templating for input / output / execution-context bindings, the trigger palette (Schedule, Webhook, Flow, Realtime) — the plugin ecosystem across warehouses (Snowflake, BigQuery), transforms (dbt), ingest (Airbyte, Fivetran), lakes (S3, GCS), and streams (Kafka, Elasticsearch), the executor / worker / queue distributed architecture on Kafka + Postgres + Elasticsearch, and the comparison against Airflow, Dagster, and Prefect that senior interviewers actually probe. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the SQL practice library →, rehearse system design against the design practice library →, and sharpen the streaming axis with the streaming practice library →.

---

On this page

• Why Kestra matters in 2026
• YAML flow syntax + task types
• Plugin ecosystem + integrations
• Executors, workers and queue architecture
• Kestra vs Airflow / Dagster / Prefect + interview signals
• Cheat sheet — Kestra recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why Kestra matters in 2026

YAML-first, polyglot, UI-driven — the orchestrator sweet spot between Airflow and Argo Workflows

The one-sentence invariant: Kestra is the open-source, YAML-first orchestrator built for polyglot data platforms — every workflow is a declarative YAML file that binds task types (Script, Sql, Http, EachParallel, Switch), trigger types (Schedule, Webhook, Flow, Realtime), and Pebble-templated bindings into a single flow definition that the executor / worker cluster runs against Kafka + Postgres + Elasticsearch; the pattern you pick in month one becomes the platform your entire team hard-codes assumptions against for years, so the choice matters far beyond "which scheduler do we use". Every senior data-engineering interviewer in 2026 asks about orchestrator choice because it separates architects who understand polyglot platform economics from candidates who have only ever run one language's scheduler.

The four axes interviewers actually probe.

• Language of authoring. Airflow flows are Python DAGs (with DAG(...) as dag: task_a >> task_b). Dagster and Prefect are also Python-first. Kestra flows are YAML files, and every task type — including Python, SQL, JavaScript, shell, and container tasks — is expressed declaratively. This makes Kestra flows language-agnostic on the authoring side while still executing polyglot code on the worker side. Interviewers open with this question because YAML-vs-Python authoring is the single biggest cultural fit decision for a data platform.
• UI experience. Airflow's UI is largely read-only — the DAG is defined in code and the UI reflects state. Kestra's UI is authoring-grade — flows can be created, edited, tested, and deployed directly from the web editor with live schema completion, no local IDE required. This is the pattern that unlocks analyst-adjacent teams (analytics engineers, BI, ops) who want to schedule things without learning Airflow's Python API.
• Plugin ecosystem. Kestra plugins are drop-in JAR files loaded by the JVM at boot — Snowflake, BigQuery, dbt, Airbyte, Fivetran, S3, GCS, Kafka, Elasticsearch, and 600+ others are first-class task types (type: io.kestra.plugin.gcp.bigquery.Query). Compared to Airflow's Python-package provider model, the JAR-plus-manifest model gives zero-conflict dependency isolation — you never fight pip for a version pin.
• Distributed execution model. Kestra's cluster is executor + worker + webserver + scheduler + indexer, backed by Kafka for the message bus and Postgres (or Elasticsearch for older releases) for state. The executor / worker split lets you scale schedulers and execution independently. This is a step function over Airflow's celery/kubernetes executor model — simpler mental model, cleaner scaling story.

The 2026 reality — Kestra sits between Airflow and Argo Workflows.

• Airflow remains the volume leader — massive community, deep Python integrations, mature operator ecosystem. The default answer when the team is Python-first and the workflow logic lives naturally in Python code. Airflow flows are Python DAGs; the UI is read-mostly; scaling is Celery, Kubernetes, or LocalExecutor depending on your patience.
• Argo Workflows is the K8s-native answer — every task is a container, every workflow a Kubernetes CRD. Fantastic when your team lives on Kubernetes and your workloads are naturally containerised; friction when you want to run a five-line Python transform without shipping a container image.
• Kestra occupies the sweet middle — YAML-declarative flows (like Argo), plugin-based first-class integrations (unlike Argo), a proper UI (unlike Argo), and language-agnostic authoring (unlike Airflow). Adopted by teams that outgrew Airflow's Python-only ceiling but did not want to K8s-native everything.
• Dagster + Prefect are the "Python-plus-modern-metadata" answers — Dagster with software-defined assets (SDA), Prefect with dynamic Python flows. Both are strong in their niche but neither offers the YAML-first, UI-authoring story that Kestra ships out of the box.

What interviewers listen for.

• Do you name the YAML flow shape — id / namespace / tasks — as the primary authoring surface, not just "some config file"? — senior signal.
• Do you distinguish "polyglot on the worker" from "polyglot on the author" — Airflow can run any language via BashOperator but the flow itself is Python? — senior signal.
• Do you name the plugin-as-JAR model and its dependency-isolation advantage over Airflow's Python-package providers? — required answer.
• Do you name the executor / worker split on Kafka + Postgres, and understand that both can scale independently? — senior signal.
• Do you describe Kestra as "a declarative flow engine with a real UI" rather than as vague "another Airflow"? — required answer.

Worked example — the four-orchestrator comparison table

Detailed explanation. The single most useful artifact for a Kestra interview is a memorised 4x4 comparison table. Every senior orchestrator discussion converges on this table within the first ten minutes; having it in your head is what separates a fluent answer from a stumbling one. Walk through building the table for a hypothetical greenfield data platform that needs to serve batch ETL, streaming enrichment, and analyst-authored schedules.

• Workload. Mixed batch (dbt + Snowflake), streaming (Kafka → S3 sink), and analyst schedules (Airbyte syncs + dashboards refresh).
• Team shape. Six data engineers (comfortable with Python), four analytics engineers (SQL + light Python), two BI analysts (SQL + YAML at most).
• Requirements. Every team should be able to author schedules without learning a new language, but code review and rollout should stay engineer-controlled.
• Constraints. Multi-cloud (AWS + GCP), self-hosted preferred over managed, container-friendly but not K8s-mandatory.

Question. Build the four-orchestrator comparison for the greenfield platform and pick the orchestrator each requirement drives you toward.

Input.

Orchestrator	Authoring surface	UI experience	Plugin model	Distributed backend
Kestra	YAML files	authoring-grade UI	drop-in JAR plugins	Kafka + Postgres
Airflow	Python DAGs	read-mostly UI	Python provider packages	Celery / K8s / Local
Dagster	Python + SDA decorators	rich, code-graph UI	Python packages	Postgres + Daemon
Prefect	Python decorators + tasks	modern SaaS-first UI	Python packages	Prefect server or Cloud

Code.

# Kestra flow — greenfield example, YAML-first
id: nightly_orders_refresh
namespace: dev.warehouse
description: "dbt run + Snowflake merge + Slack notify"

tasks:
  - id: dbt_run
    type: io.kestra.plugin.dbt.cli.DbtCLI
    commands:
      - dbt run --select tag:orders
    profiles: |
      warehouse:
        target: prod
        outputs:
          prod:
            type: snowflake
            account: "{{ secret('SNOWFLAKE_ACCOUNT') }}"
            user: "{{ secret('SNOWFLAKE_USER') }}"
            password: "{{ secret('SNOWFLAKE_PASSWORD') }}"
            role: TRANSFORMER
            database: PROD
            warehouse: TRANSFORM_WH
            schema: ANALYTICS

  - id: notify_slack
    type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
    url: "{{ secret('SLACK_URL') }}"
    payload: |
      {"text": "Nightly orders refresh complete — {{ execution.state.current }}"}

triggers:
  - id: daily
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 2 * * *"

Step-by-step explanation.

1. The comparison table forces the four axes into a single view: authoring surface, UI experience, plugin model, and distributed backend. Kestra wins on YAML-first authoring (analysts can read the flow without learning Python), draws with Dagster on UI richness, and uniquely offers a JAR-based plugin model that keeps dependency isolation clean.
2. Airflow is the default answer only when the team is Python-first and every scheduler user is comfortable with the DAG API — analytics engineers and BI folks reading with DAG(...) as dag: incurs real cognitive cost.
3. Dagster is the strongest answer when the platform is asset-centric — you want a materialised lineage graph, per-asset checks, and SDA-first thinking. If most of your workflows are "materialise this table if inputs changed," Dagster is a fit.
4. Prefect is the strongest answer when flows are highly dynamic — the flow topology depends on runtime data, not a static graph. Prefect's task decorators let you emit tasks conditionally and inspect them at runtime.
5. The greenfield answer for our scenario is Kestra: it gives you YAML-first authoring (analytics engineers and BI can read and edit flows), an authoring-grade UI (BI folks do not need a local IDE), JAR-based plugins (Snowflake, dbt, S3, Slack out of the box), and Kafka-backed execution (scales cleanly without a Celery-vs-K8s religious war).

Output.

Requirement	Recommended orchestrator	Why
Analyst-authored schedules	Kestra	YAML + UI editor lets non-Python users author flows
Deep Python operators	Airflow	Python-first ecosystem, most operators
Asset-lineage first	Dagster	SDA + materialisation graph out of the box
Highly dynamic flows	Prefect	Runtime graph, task decorators, dynamic dispatch
Multi-cloud + polyglot tasks	Kestra	YAML + JAR plugins keep dependency isolation clean

Rule of thumb. Never pick an orchestrator based on "the team already uses Python." Pick it on (authoring surface × UI × plugin model × distributed backend) — the four axes. Write the table on a whiteboard first; the orchestrator choice falls out of the constraints.

Worked example — what interviewers actually probe on Kestra

Detailed explanation. The senior data-engineering Kestra interview has a predictable structure: the interviewer opens with an ambiguous question ("what would you use for the orchestration layer?"), then progressively narrows to test whether you understand YAML flows, plugins, triggers, and the executor / worker split. The candidates who name the pattern in sentence one score highest; the candidates who describe "just another Airflow" score lowest. Walk through the interview grading rubric.

• Ambiguous opener. "Walk me through how you'd design the orchestration layer for a new data platform." — invites you to name Kestra and the YAML-first model.
• Follow-up 1. "How do teams author flows?" — probes UI and YAML authoring.
• Follow-up 2. "How do you run a dbt build alongside a Python transform in the same flow?" — probes plugin model and polyglot tasks.
• Follow-up 3. "What happens when the flow scheduler dies?" — probes executor / worker split.
• Follow-up 4. "How does Kestra compare to Airflow?" — probes the four-axis comparison.

Question. Draft a 5-minute senior Kestra answer that covers all four axes without waiting to be asked.

Input.

Interview signal	Weak answer	Senior answer
Orchestrator named	"we'd use Airflow because everyone knows it"	"Kestra — YAML-first, polyglot, UI-driven — Airflow if the team is Python-only"
Authoring surface	"you write a DAG"	"YAML flows with id, namespace, tasks; Pebble templating"
Plugin model	"there are operators"	"drop-in JAR plugins loaded at boot; type: io.kestra.plugin.gcp.bigquery.Query"
Execution model	"there's a scheduler and workers"	"executor + worker split on Kafka; Postgres for state; Elasticsearch for history"
Trigger palette	"cron schedules"	"Schedule, Webhook, Flow (event-driven), Realtime (Kafka consumer)"

Code.

Senior Kestra answer template (5 minutes)
=========================================

Minute 1 — name the orchestrator up front
  "I'd default to Kestra — a YAML-first, polyglot orchestrator with an
   authoring-grade UI. Airflow if the team is Python-only; Dagster if
   we need asset lineage; Prefect if flows are highly dynamic."

Minute 2 — authoring model
  "Every flow is a YAML file with `id`, `namespace`, and a list of
   `tasks`. Each task has a `type:` pointing at a plugin class —
   `io.kestra.plugin.gcp.bigquery.Query`, `io.kestra.plugin.dbt.cli.DbtCLI`,
   `io.kestra.plugin.scripts.python.Script`. Pebble templating binds
   inputs, outputs, and execution context — `{{ execution.startDate }}`,
   `{{ inputs.date }}`, `{{ outputs.step1.uri }}`."

Minute 3 — plugin ecosystem
  "Plugins are drop-in JAR files loaded at boot — 600+ plugins covering
   Snowflake, BigQuery, dbt, Airbyte, Fivetran, S3, GCS, Kafka,
   Elasticsearch, Slack, Postgres. Custom plugins are Java classes
   annotated with `@Plugin` — dependency isolation is JVM-level, no
   pip conflicts."

Minute 4 — distributed execution
  "Cluster = executor + worker + webserver + scheduler + indexer,
   backed by Kafka (message bus) + Postgres (state) + optional
   Elasticsearch (search/history). Executor schedules tasks onto Kafka
   topics; workers consume by `worker.group` tag; scaling is horizontal
   per component."

Minute 5 — triggers + governance
  "Trigger types: Schedule (cron), Webhook (HTTP-triggered), Flow
   (event-driven from other flows), Realtime (Kafka consumer). Every
   flow is versioned in git; UI edits push commits; RBAC is
   namespace-scoped."

Step-by-step explanation.

1. Minute 1 is the crucial framing. Naming Kestra immediately — "YAML-first, polyglot, UI-driven" — signals you're a decision-maker who understands the orchestrator landscape, not a task-runner who reaches for the tool the last team used. Weak candidates dive into "well, we've always used Airflow…" before naming any decision criteria.
2. Minute 2 addresses the authoring axis. Naming the id / namespace / tasks shape and Pebble templating up front demonstrates you have actually authored a Kestra flow, not just read the marketing page. Interviewers love this signal because it's cheap to fake unless you've actually run the tool.
3. Minute 3 is the plugin probe. Naming JAR-based plugins and contrasting with Airflow's pip-based providers shows you understand the operational win — dependency isolation at the JVM level rather than the pip level. The @Plugin annotation for custom plugins is a senior detail.
4. Minute 4 covers the distributed execution model. The executor / worker split on Kafka is the specific detail that separates Kestra from Airflow's Celery/Kubernetes executor debate. Naming Postgres for state and Elasticsearch for history shows you understand the backend deployment story.
5. Minute 5 covers triggers and governance. Naming all four trigger types (Schedule, Webhook, Flow, Realtime) — especially Flow (event-driven) and Realtime (Kafka consumer) — signals you understand Kestra's differentiators over cron-only schedulers.

Output.

Grading criterion	Weak score	Senior score
Names Kestra in minute 1	rare	mandatory
Names YAML flow shape	rare	required
Names JAR plugin model	rare	senior signal
Names executor/worker split	rare	senior signal
Names all four trigger types	rare	senior signal

Rule of thumb. The senior Kestra answer is a 5-minute monologue that covers all four axes — authoring, plugin, execution, triggers — without waiting for the follow-ups. Rehearse it once; deploy it every time.

Worked example — the "pick the orchestrator" decision tree

Detailed explanation. Given a new platform brief, the senior architect runs a 4-question decision tree in their head. Codifying the tree makes the interview answer reproducible: any interviewer can hand you a scenario and you can walk the tree out loud. Walk through the tree with three canonical scenarios: a Python-only team scaling out of a monolith, a multi-team platform serving analysts and engineers, and a K8s-native shop doing streaming ETL.

• Q1. Is the team Python-only or truly polyglot? → Python-only = Airflow / Dagster / Prefect; polyglot = Kestra.
• Q2. Do non-engineers need to author schedules? → yes = Kestra (YAML + UI); no = any Python orchestrator.
• Q3. Is asset lineage a first-class requirement? → yes = Dagster; no = anything else.
• Q4. Are flows highly dynamic (runtime topology)? → yes = Prefect; no = static-graph orchestrators.
• Q5 (K8s branch). Do you already run everything on K8s and want containers-first? → yes = Argo Workflows (Kestra also works on K8s but is not K8s-first).

Question. Walk the decision tree for the three scenarios and record the orchestrator each ends up with.

Input.

Scenario	Q1 (polyglot?)	Q2 (non-engineer authors?)	Q3 (asset lineage?)	Q5 (K8s-first?)
Python monolith scaling out	no	no	no	no
Multi-team analyst platform	yes	yes	no	no
K8s-native streaming ETL	yes	no	no	yes

Code.

# Decision-tree helper (illustrative)
def pick_orchestrator(polyglot: bool,
                       non_engineer_authors: bool,
                       asset_lineage: bool,
                       dynamic_flows: bool,
                       k8s_native: bool) -> str:
    """Return the primary orchestrator for a new platform."""
    if k8s_native and polyglot and not non_engineer_authors:
        return "Argo Workflows (Kestra as fallback if plugins matter)"
    if non_engineer_authors:
        return "Kestra (YAML + UI editor)"
    if polyglot:
        return "Kestra"
    if asset_lineage:
        return "Dagster"
    if dynamic_flows:
        return "Prefect"
    return "Airflow"


# Walk the three scenarios
print(pick_orchestrator(False, False, False, False, False))
# → 'Airflow'

print(pick_orchestrator(True, True, False, False, False))
# → 'Kestra (YAML + UI editor)'

print(pick_orchestrator(True, False, False, False, True))
# → 'Argo Workflows (Kestra as fallback if plugins matter)'

Step-by-step explanation.

1. Scenario 1 — a Python monolith scaling out. Everyone on the team writes Python; no analysts author schedules; asset lineage is nice-to-have but not first-class. The tree short-circuits at Q1 (not polyglot) and Q3 (no asset lineage) → Airflow. This is the modern default for Python-first teams.
2. Scenario 2 — a multi-team analyst platform. Engineers write Python; analysts write SQL; BI folks want a UI. Q1 (polyglot) = yes; Q2 (non-engineers author) = yes → Kestra. This is the sweet-spot scenario for Kestra adoption in 2026.
3. Scenario 3 — a K8s-native streaming ETL shop. Everything is containers; the team lives on K8s; Argo Workflows is the K8s-native answer. Q5 fires and Argo wins — but Kestra remains a viable fallback if the team wants first-class plugin integrations (Snowflake, dbt) that Argo does not have.
4. The parallel Q3 (Dagster) and Q4 (Prefect) branches are orthogonal to the Kestra decision. Asset lineage or highly dynamic flows can outweigh the other axes; when they do, Dagster and Prefect win despite Kestra's broader UI story.
5. If none of Q1-Q4 pass a clear winner, the tie-break is "which orchestrator does the team already know?" — the operational cost of learning a new tool is real, and switching costs matter more than any single feature.

Output.

Scenario	Primary orchestrator	Notes
Python monolith scaling out	Airflow	Python-first, largest community
Multi-team analyst platform	Kestra	YAML + UI unblocks non-engineer authors
K8s-native streaming ETL	Argo Workflows	K8s-first; Kestra as plugin-rich fallback

Rule of thumb. The five-question decision tree is a whiteboard-friendly answer. Practice walking it end-to-end so an interviewer can hand you any scenario and get an orchestrator name in under 60 seconds.

Senior interview question on orchestrator selection

A senior interviewer often opens with: "You are joining a data platform team that runs a mix of Python transforms, dbt models, and analyst-authored SQL scripts. The current setup is 200 crontabs on 12 EC2 instances plus a broken Airflow 1.x install. Walk me through the orchestrator you'd migrate to, the migration plan, and the interview signals you'd probe when hiring the team to run it."

Solution Using Kestra with a phased crontab migration, plugin-first onboarding, and executor / worker split

# 1. Kestra namespace layout — one namespace per team, one flow per job family
# namespace: dev.warehouse
# namespace: dev.analytics
# namespace: dev.streaming

# 2. Reference flow — replaces one legacy crontab line
id: legacy_orders_extract
namespace: prod.warehouse
description: "Nightly Postgres → S3 extract, replaces the 02:00 crontab on host-04"

inputs:
  - id: run_date
    type: DATE
    defaults: "{{ execution.startDate | date('yyyy-MM-dd') }}"

tasks:
  - id: extract_pg
    type: io.kestra.plugin.jdbc.postgresql.Query
    url: "jdbc:postgresql://db-primary.internal:5432/production"
    username: "{{ secret('PG_USER') }}"
    password: "{{ secret('PG_PASSWORD') }}"
    sql: |
      SELECT id, customer_id, total_cents, status, updated_at
      FROM   public.orders
      WHERE  updated_at::date = '{{ inputs.run_date }}'
    fetchType: STORE

  - id: upload_s3
    type: io.kestra.plugin.aws.s3.Upload
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
    region: us-east-1
    bucket: acme-lake
    key: "raw/orders/{{ inputs.run_date }}.parquet"
    from: "{{ outputs.extract_pg.uri }}"

  - id: notify
    type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
    url: "{{ secret('SLACK_URL') }}"
    payload: |
      {"text": "orders extract for {{ inputs.run_date }} → s3://acme-lake/raw/orders/"}

triggers:
  - id: nightly
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 2 * * *"
    inputs:
      run_date: "{{ trigger.date | date('yyyy-MM-dd') }}"

# 3. docker-compose.yml — standalone dev, then split into executor / worker for prod
version: "3.8"
services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: kestra
      POSTGRES_PASSWORD: kestra
      POSTGRES_DB: kestra
    volumes:
      - postgres-data:/var/lib/postgresql/data

  kestra:
    image: kestra/kestra:latest
    command: server standalone
    environment:
      KESTRA_CONFIGURATION: |
        datasources:
          postgres:
            url: jdbc:postgresql://postgres:5432/kestra
            driverClassName: org.postgresql.Driver
            username: kestra
            password: kestra
        kestra:
          storage:
            type: local
            local:
              base-path: /app/storage
          repository:
            type: postgres
          queue:
            type: postgres
    ports:
      - "8080:8080"
    depends_on:
      - postgres
    volumes:
      - kestra-storage:/app/storage

volumes:
  postgres-data:
  kestra-storage:

Step-by-step trace.

Phase	Before (crontab chaos)	After (Kestra)
Discovery	grep -r crontab /etc/cron.d on 12 hosts	UI shows every flow in one namespace tree
Authoring	ssh in, edit crontab, hope	YAML file in git, PR-reviewed, deployed via UI or CLI
Observability	tail -f log files on hosts	UI shows execution graph, per-task logs, retries
Retry semantics	manual re-run	retry: block per task; automatic exponential backoff
Secret handling	plaintext in scripts	{{ secret('KEY') }} from Kestra secret store
Scaling	add more crontabs on more hosts	scale worker replicas; assign worker.group

After the migration, the 200 legacy crontabs live as 200 YAML flows under prod.legacy.* namespaces; every flow is version-controlled in git; every failure sends a Slack ping; every retry is bounded. The two half-broken Airflow 1.x installs are gone; the Airflow 2.x migration that never happened is now moot.

Output:

Metric	Before	After
Flow discovery	200 crontabs across 12 hosts	200 flows in Kestra UI, one namespace tree
Authoring bar	shell + cron syntax	YAML + Pebble templating + UI editor
Retry story	manual re-run	declarative retry: per task
Secret story	plaintext	{{ secret('KEY') }} with backend of choice
Observability	grep logs on host	UI graph + per-task logs + execution history
Scaling	add EC2 instances	scale worker replicas horizontally

Why this works — concept by concept:

• YAML flow shape — id, namespace, tasks, triggers is the entire contract. Every crontab line becomes one YAML file; the namespace tree gives you organisational structure without a database schema; the triggers list makes schedules first-class rather than second-class shell wrappers.
• Pebble templating — {{ execution.startDate }}, {{ inputs.run_date }}, {{ outputs.extract_pg.uri }}, {{ secret('PG_USER') }} — bindings that turn a static flow into a parameterised one without leaking Python-specific syntax. Analysts can read Pebble; Jinja fans recognise it instantly.
• Plugin task types — io.kestra.plugin.jdbc.postgresql.Query, io.kestra.plugin.aws.s3.Upload, io.kestra.plugin.notifications.slack.SlackIncomingWebhook are drop-in JAR-loaded task types. No pip install; no Python dependency conflicts; no "which Airflow provider version handles this parameter" investigation.
• Standalone → distributed migration — kestra server standalone runs everything in one JVM for dev; the production cluster splits into executor + worker + webserver + scheduler + indexer, sharing Postgres and Kafka. The dev-to-prod migration is a config change, not a rewrite.
• Cost — one Postgres (state) + one Kafka (queue) + N worker replicas + one executor + one webserver = ~4 vCPU baseline. Compared to 12 EC2 instances running crontabs plus a broken Airflow install, this is dramatically cheaper and the flows are discoverable. Net O(1) per new flow (write YAML, git push) versus O(host-count) for crontabs. The eliminated cost is the "which crontab actually runs the job we're debugging" investigation.

SQL
Topic — sql
SQL orchestration and pipeline problems

Practice →

Design Topic — design Design problems on data orchestration

Practice →

---

2. YAML flow syntax + task types

id / namespace / tasks is the entire authoring contract — Pebble templating and typed task classes fill in the rest

The mental model in one line: a kestra workflow is a YAML file with three mandatory top-level keys — id (a unique identifier within the namespace), namespace (a dot-separated organisational path like prod.warehouse), and tasks (an ordered list of typed task blocks) — plus optional inputs, outputs, triggers, errors, labels, and retry; every task carries a type: that points at a Java plugin class, and Pebble templating ({{ execution.startDate }}, {{ inputs.foo }}, {{ outputs.step1.uri }}) is the binding language that turns a static flow into a parameterised one. Every senior Kestra flow is built from this one contract; no exotic YAML magic beyond the standard block scalars and lists.

The four building blocks every Kestra flow uses.

• The top-level keys. id + namespace uniquely identify a flow; tasks is the ordered pipeline; triggers declares when the flow runs; inputs declares parameters (typed — STRING, DATE, INT, JSON, FILE); outputs optionally re-exports task outputs at flow level; labels are searchable metadata; errors is a nested pipeline that runs on failure. Every flow uses id, namespace, and tasks; the rest are opt-in.
• The task types. Every task has a type: field pointing at a Java class name — io.kestra.plugin.scripts.python.Script, io.kestra.plugin.jdbc.postgresql.Query, io.kestra.plugin.core.http.Request, io.kestra.plugin.core.flow.EachParallel, io.kestra.plugin.core.flow.Switch. The type determines which properties the task accepts (schema validation happens in the UI editor as you type).
• Pebble templating. Kestra uses Pebble — a Jinja-like templating engine — for all bindings. {{ execution.startDate }} is the flow's start timestamp; {{ inputs.date }} is the value of an input named date; {{ outputs.step1.uri }} is a URI produced by an upstream task; {{ secret('KEY') }} reads from the Kestra secret store; {{ vars.foo }} reads from namespace-scoped variables.
• The trigger palette. Triggers live under the top-level triggers: key and are also typed. io.kestra.plugin.core.trigger.Schedule for cron; io.kestra.plugin.core.trigger.Webhook for HTTP-triggered; io.kestra.plugin.core.trigger.Flow for event-driven from other flows; io.kestra.plugin.core.trigger.Realtime for streaming (e.g. Kafka consumer).

The built-in task type catalog — the 5 you use every day.

• io.kestra.plugin.scripts.python.Script. Runs a Python script, either inline in a script: block or from a file. Handles requirements (beforeCommands: [pip install requests]), captures stdout/stderr as outputs, and integrates with Kestra's file store via outputFiles: [...].
• io.kestra.plugin.jdbc.postgresql.Query. Runs SQL against Postgres (analog task types exist for MySQL, Snowflake, BigQuery, Redshift, Clickhouse). Returns either a fetched rowset (fetchType: FETCH) or a stored file (fetchType: STORE) for streaming downstream.
• io.kestra.plugin.core.http.Request. Issues an HTTP call; captures response body, headers, and status. Supports OAuth2, basic auth, and templated headers. The "call any REST API" primitive.
• io.kestra.plugin.core.flow.EachParallel / EachSequential. Iterates over a list of values (from an upstream output or an input) and runs a sub-pipeline of tasks per value. EachParallel runs iterations concurrently; EachSequential runs them one at a time. This is Kestra's answer to Airflow's expand() dynamic task mapping.
• io.kestra.plugin.core.flow.Switch. Branches on a Pebble expression. Each case is a nested tasks: list. Runs exactly one case per execution. Kestra's answer to Airflow's BranchPythonOperator.

Pebble templating — the four bindings you will use every flow.

• {{ execution.startDate }} — the flow execution's start timestamp (ISO-8601). Filters exist for formatting: {{ execution.startDate | date('yyyy-MM-dd') }}.
• {{ inputs.<name> }} — the value of an input declared under the top-level inputs: key. Typed inputs (DATE, INT, JSON, FILE) are automatically parsed.
• {{ outputs.<taskId>.<field> }} — a field from an upstream task's output. The available fields depend on the task type; the UI editor auto-completes them.
• {{ secret('<key>') }} / {{ vars.<key> }} — secret and variable lookups. Secrets are backend-pluggable (JDBC store, HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager); vars are namespace-scoped.

The trigger palette — four kinds of "run" event.

• Schedule — cron expression, timezone-aware. The bread-and-butter trigger for batch pipelines. Supports late-data catchup and backfill.
• Webhook — Kestra exposes a URL like /api/v1/executions/webhook/<flow>/<random-key>; any HTTP POST triggers a flow execution. The webhook body becomes {{ trigger.body }}.
• Flow — one flow's completion triggers another flow. Fields: states: [SUCCESS, FAILED], namespace:, flowId:. This is the event-driven fan-out primitive.
• Realtime — a persistent consumer (Kafka, Pulsar, Google Pub/Sub) that spawns one execution per message. This turns Kestra into a lightweight streaming processor.

Common interview probes on YAML flow syntax.

• "How is a Kestra flow structured?" — required answer: id, namespace, tasks at top level; each task has a type: pointing at a plugin class.
• "How do you pass values between tasks?" — Pebble bindings: {{ outputs.step1.uri }} or {{ outputs.step1.rows }}.
• "How do you templatise the flow with an input date?" — declare inputs: at flow level; reference as {{ inputs.date }}; feed it from the trigger.
• "How do you iterate a task over a list?" — EachParallel or EachSequential with a value: (Pebble expression evaluating to a list) and a nested tasks: block.

Worked example — hello-world flow with inputs and Pebble bindings

Detailed explanation. The canonical Kestra "hello world" flow: one input, two tasks — one Script that generates a value, one Log that prints it — bound together via a Pebble output reference. Walk through every line to see the entire authoring contract in ~20 lines of YAML.

• Input. A name input of type STRING with a default value.
• Task 1. A Python script that echoes a JSON output.
• Task 2. A Log task that references the Python task's output via Pebble.

Question. Write the hello-world flow with one input, one Python task, and one Log task that pulls its message from the Python task's output.

Input.

Component	Value
Namespace	dev.demo
Flow id	hello_world
Input	name (STRING, default 'World')
Task 1	Python Script, echoes greeting
Task 2	Log, references task 1's output

Code.

id: hello_world
namespace: dev.demo
description: "The canonical Kestra hello-world flow"

inputs:
  - id: name
    type: STRING
    defaults: "World"
    description: "Who to greet"

tasks:
  - id: build_greeting
    type: io.kestra.plugin.scripts.python.Script
    beforeCommands:
      - pip install kestra
    script: |
      import json
      from kestra import Kestra

      name = "{{ inputs.name }}"
      message = f"Hello, {name}! It is {{ '{{' }} execution.startDate | date('yyyy-MM-dd HH:mm') {{ '}}' }}."

      # Kestra client sends the value as a named output field
      Kestra.outputs({"greeting": message})

  - id: log_greeting
    type: io.kestra.plugin.core.log.Log
    message: "{{ outputs.build_greeting.vars.greeting }}"

  - id: log_context
    type: io.kestra.plugin.core.log.Log
    message: |
      execution.id       = {{ execution.id }}
      execution.startDate = {{ execution.startDate }}
      inputs.name        = {{ inputs.name }}
      namespace          = {{ flow.namespace }}

Step-by-step explanation.

1. The top-level keys id: hello_world and namespace: dev.demo uniquely identify the flow. The description: is displayed in the UI. This is the minimal metadata every flow needs.
2. The inputs: block declares one typed input named name with a default. On execution the UI presents a form field; the CLI accepts --input name=Alice; scheduled runs use the default. Typed inputs (STRING, DATE, INT, JSON, FILE) are validated before the flow starts.
3. The build_greeting task uses io.kestra.plugin.scripts.python.Script — the built-in Python task type. beforeCommands: runs before the script (typically pip install); the script: block is inline Python. The Kestra.outputs({"greeting": message}) call sends a named output to the executor via a lightweight IPC channel; downstream tasks reference it via Pebble.
4. The log_greeting task uses io.kestra.plugin.core.log.Log — the simplest task type, just writes a message to the execution log. Its message: field is a Pebble template: {{ outputs.build_greeting.vars.greeting }} reads the greeting field from the upstream task's vars map (that's where Kestra.outputs({...}) values land).
5. The log_context task shows the four most common execution-context bindings: {{ execution.id }} (unique per run), {{ execution.startDate }} (ISO-8601 timestamp), {{ inputs.name }} (the input value), and {{ flow.namespace }} (metadata about the flow itself). These are your debugging staples.

Output.

Task	Output shape	Downstream reference
build_greeting	{ vars: { greeting: 'Hello, World! It is 2026-07-21 02:00.' } }	{{ outputs.build_greeting.vars.greeting }}
log_greeting	log-only, no downstream output	(none)
log_context	log-only, no downstream output	(none)

Rule of thumb. For any new Kestra flow, start with id, namespace, tasks. Add inputs: when you need parameters; add triggers: when you need scheduling; add outputs: when downstream flows need this flow's values. Skip anything the flow does not use — Kestra's schema validation flags missing required fields loudly.

Worked example — Postgres extract with typed inputs and Pebble date binding

Detailed explanation. A more realistic flow: a scheduled daily extract from Postgres, parameterised by a run date, with the SQL query templated via Pebble to read the correct partition. Walk through the flow to see how inputs, triggers, plugin tasks, and Pebble compose for a typical batch job.

• Input. A run_date of type DATE with a default computed from the execution date.
• Task 1. Postgres Query that reads yesterday's orders based on the input date.
• Task 2. S3 Upload that writes the resulting file with a date-partitioned key.
• Trigger. Nightly at 02:00 UTC, passing yesterday's date as run_date.

Question. Write the daily Postgres → S3 extract flow with a typed DATE input and a Schedule trigger that computes the run date automatically.

Input.

Component	Value
Namespace	prod.warehouse
Flow id	daily_orders_extract
Input	run_date (DATE)
Trigger	Schedule, 0 2 * * *, feeds `{{ trigger.date
Task 1	PostgreSQL Query, fetchType STORE
Task 2	S3 Upload, partitioned key

Code.
{% raw %}

id: daily_orders_extract
namespace: prod.warehouse
description: "Nightly Postgres → S3 extract for the orders fact table"

labels:
  team: warehouse
  tier: production

inputs:
  - id: run_date
    type: DATE
    defaults: "{{ execution.startDate | date('yyyy-MM-dd') }}"
    description: "Partition date; defaults to execution date"

tasks:
  - id: extract_orders
    type: io.kestra.plugin.jdbc.postgresql.Query
    url: "jdbc:postgresql://db-primary.internal:5432/production"
    username: "{{ secret('PG_USER') }}"
    password: "{{ secret('PG_PASSWORD') }}"
    sql: |
      SELECT id,
             customer_id,
             total_cents,
             status,
             created_at,
             updated_at
      FROM   public.orders
      WHERE  updated_at::date = '{{ inputs.run_date }}'
      ORDER  BY updated_at ASC
    fetchType: STORE

  - id: upload_to_s3
    type: io.kestra.plugin.aws.s3.Upload
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
    region: us-east-1
    bucket: acme-lake
    key: "raw/orders/dt={{ inputs.run_date }}/orders.parquet"
    from: "{{ outputs.extract_orders.uri }}"

  - id: notify_success
    type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
    url: "{{ secret('SLACK_URL') }}"
    payload: |
      {
        "text": "Orders extract complete — run_date={{ inputs.run_date }} → s3://acme-lake/raw/orders/dt={{ inputs.run_date }}/",
        "attachments": [{
          "title": "execution",
          "text": "{{ execution.id }}"
        }]
      }

triggers:
  - id: nightly
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 2 * * *"
    timezone: UTC
    inputs:
      run_date: "{{ trigger.date | date('yyyy-MM-dd', existingFormat='yyyy-MM-dd HH:mm:ssX') }}"

errors:
  - id: notify_failure
    type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
    url: "{{ secret('SLACK_URL') }}"
    payload: |
      {
        "text": "❌ Orders extract failed — run_date={{ inputs.run_date }} — {{ execution.id }}"
      }

Step-by-step explanation.

1. The labels: block adds searchable metadata — team: warehouse lets the platform team filter for all warehouse flows in the UI. Labels do not affect execution; they are governance metadata.
2. The inputs: block declares run_date as a DATE input with a default computed via Pebble. On scheduled runs the trigger overrides the default; on manual runs the operator sees a UI date picker; on API-triggered runs the client passes the value.
3. The extract_orders task uses the Postgres plugin's Query task type. fetchType: STORE writes the result to Kestra's internal file store as a Parquet file (default) and exposes it as {{ outputs.extract_orders.uri }} — a temporary URI the downstream task can consume. Alternative: fetchType: FETCH returns rows in memory as outputs.extract_orders.rows, useful for small result sets.
4. The upload_to_s3 task uses the AWS plugin's Upload task type. The from: field consumes the upstream URI via Pebble; the key: field builds a date-partitioned S3 key using the input. This composition is the entire "extract + load" pattern — no intermediate files, no manual data transfer.
5. The errors: block runs only on flow failure — a Slack notification that includes the failing execution's ID. This is Kestra's answer to Airflow's on_failure_callback; declarative rather than callback-based.

Output.

Task	Output field	Example value
extract_orders	uri	kestra:///storage/…/orders.ion
extract_orders	size	12345 (bytes)
upload_to_s3	(no output; side-effect only)	s3://acme-lake/raw/orders/dt=2026-07-20/orders.parquet
notify_success	(no output; side-effect only)	Slack message posted

Rule of thumb. For any scheduled batch flow, use a typed DATE input with a default computed from execution.startDate, then override the default in the trigger with trigger.date. This makes the flow work identically for scheduled runs and manual backfills — same code path, different input value.

Worked example — parallel iteration with EachParallel and Switch branching

Detailed explanation. Real-world flows need dynamic fan-out and conditional branching. Kestra provides EachParallel (concurrent iteration) and Switch (branch on Pebble expression). Walk through a flow that lists all Postgres tables in a namespace, extracts each one in parallel, and branches per table on whether it should be uploaded to S3 or GCS.

• Task 1. List tables via a metadata query.
• Task 2. EachParallel over the table list — one branch per table.
• Task 3 (inside branch). Switch on the table name — S3 upload for raw.*, GCS upload for curated.*.

Question. Write the parallel-per-table flow with EachParallel iteration and Switch branching inside each iteration.

Input.

Component	Value
Tables	list of TEXT names from information_schema
Concurrency	one branch per table, running in parallel
Branching	Switch on table prefix (raw.* vs curated.*)
Destinations	S3 for raw, GCS for curated

Code.

id: parallel_multi_table_extract
namespace: prod.warehouse
description: "List tables, extract each in parallel, route by prefix"

tasks:
  - id: list_tables
    type: io.kestra.plugin.jdbc.postgresql.Query
    url: "jdbc:postgresql://db-primary.internal:5432/production"
    username: "{{ secret('PG_USER') }}"
    password: "{{ secret('PG_PASSWORD') }}"
    sql: |
      SELECT schemaname || '.' || tablename AS table_name
      FROM   pg_tables
      WHERE  schemaname IN ('raw', 'curated')
      ORDER  BY table_name
    fetchType: FETCH

  - id: per_table
    type: io.kestra.plugin.core.flow.EachParallel
    concurrent: 8               # cap on parallel iterations
    value: "{{ outputs.list_tables.rows | map(attribute='table_name') | list }}"
    tasks:
      - id: extract_one
        type: io.kestra.plugin.jdbc.postgresql.Query
        url: "jdbc:postgresql://db-primary.internal:5432/production"
        username: "{{ secret('PG_USER') }}"
        password: "{{ secret('PG_PASSWORD') }}"
        sql: "SELECT * FROM {{ taskrun.value }}"
        fetchType: STORE

      - id: route_by_prefix
        type: io.kestra.plugin.core.flow.Switch
        value: "{{ taskrun.value | split('.') | first }}"
        cases:
          raw:
            - id: upload_raw
              type: io.kestra.plugin.aws.s3.Upload
              accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
              secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
              region: us-east-1
              bucket: acme-lake-raw
              key: "{{ taskrun.value }}/{{ execution.startDate | date('yyyy-MM-dd') }}.parquet"
              from: "{{ outputs.extract_one.uri }}"
          curated:
            - id: upload_curated
              type: io.kestra.plugin.gcp.gcs.Upload
              serviceAccount: "{{ secret('GCP_SA_JSON') }}"
              bucket: acme-curated
              to: "gs://acme-curated/{{ taskrun.value }}/{{ execution.startDate | date('yyyy-MM-dd') }}.parquet"
              from: "{{ outputs.extract_one.uri }}"
        defaults:
          - id: unknown_prefix
            type: io.kestra.plugin.core.log.Log
            message: "Unknown table prefix for {{ taskrun.value }} — skipped"
            level: WARN

Step-by-step explanation.

1. The list_tables task queries pg_tables and returns rows in memory (fetchType: FETCH). This is safe for a table-list query — a few hundred rows at most, not millions. The result exposes {{ outputs.list_tables.rows }} as a list of dicts to downstream Pebble expressions.
2. The per_table task is EachParallel — Kestra's dynamic fan-out primitive. The value: Pebble expression extracts the table names into a list ({{ outputs.list_tables.rows | map(attribute='table_name') | list }}); concurrent: 8 caps the parallel execution at 8 iterations at a time (protects the source Postgres from N-way saturation).
3. Inside each iteration, the loop variable is exposed as {{ taskrun.value }} — the current table name. extract_one uses it in the SQL to query the specific table; the STORE fetchType writes the result to a fresh temporary URI per iteration.
4. The route_by_prefix task is Switch — it evaluates the value: expression ({{ taskrun.value | split('.') | first }} extracts the schema prefix), matches against the cases: keys, and runs the matching branch. Two cases (raw, curated) route to S3 or GCS; the defaults: branch logs a warning if the prefix is unknown.
5. The composition — one flow, one YAML file — handles a variable number of tables per run, per-table branching, and per-branch cloud destinations. Airflow's equivalent (dynamic task mapping + BranchPythonOperator + Cloud provider operators) is ~3× more Python code and requires deeper knowledge of Airflow's XCom system.

Output.

Iteration	taskrun.value	Switch branch	Destination
1	raw.orders	raw	s3://acme-lake-raw/raw.orders/2026-07-21.parquet
2	raw.customers	raw	s3://acme-lake-raw/raw.customers/2026-07-21.parquet
3	curated.orders_fact	curated	gs://acme-curated/curated.orders_fact/2026-07-21.parquet
4	curated.customer_dim	curated	gs://acme-curated/curated.customer_dim/2026-07-21.parquet

Rule of thumb. For any dynamic fan-out use EachParallel with an explicit concurrent: cap — leaving it unbounded is how you accidentally launch 500 concurrent Postgres queries. For any per-value branching use Switch with a defaults: branch — an unmatched value should never silently pass through.

Senior interview question on YAML flow syntax

A senior interviewer might ask: "Write a Kestra flow that lists all Postgres tables in the sales schema, extracts each one in parallel (max 4 concurrent), uploads the results to S3 with a date-partitioned key, and sends a Slack notification with the per-table row counts. Walk me through every Pebble binding, every task type, and how error handling composes."

Solution Using EachParallel + PostgreSQL Query + S3 Upload + templated Slack notification

id: sales_multi_table_extract
namespace: prod.warehouse
description: "Parallel per-table extract for the sales schema"

labels:
  team: warehouse
  tier: production
  domain: sales

inputs:
  - id: run_date
    type: DATE
    defaults: "{{ execution.startDate | date('yyyy-MM-dd') }}"

tasks:
  - id: list_tables
    type: io.kestra.plugin.jdbc.postgresql.Query
    url: "jdbc:postgresql://db-primary.internal:5432/production"
    username: "{{ secret('PG_USER') }}"
    password: "{{ secret('PG_PASSWORD') }}"
    sql: |
      SELECT tablename
      FROM   pg_tables
      WHERE  schemaname = 'sales'
      ORDER  BY tablename
    fetchType: FETCH

  - id: per_table
    type: io.kestra.plugin.core.flow.EachParallel
    concurrent: 4
    value: "{{ outputs.list_tables.rows | map(attribute='tablename') | list }}"
    tasks:
      - id: extract
        type: io.kestra.plugin.jdbc.postgresql.Query
        url: "jdbc:postgresql://db-primary.internal:5432/production"
        username: "{{ secret('PG_USER') }}"
        password: "{{ secret('PG_PASSWORD') }}"
        sql: "SELECT * FROM sales.{{ taskrun.value }}"
        fetchType: STORE
        store: true

      - id: upload
        type: io.kestra.plugin.aws.s3.Upload
        accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
        secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
        region: us-east-1
        bucket: acme-lake
        key: "raw/sales/{{ taskrun.value }}/dt={{ inputs.run_date }}/data.parquet"
        from: "{{ outputs.extract.uri }}"

      - id: count_rows
        type: io.kestra.plugin.jdbc.postgresql.Query
        url: "jdbc:postgresql://db-primary.internal:5432/production"
        username: "{{ secret('PG_USER') }}"
        password: "{{ secret('PG_PASSWORD') }}"
        sql: "SELECT count(*) AS n FROM sales.{{ taskrun.value }}"
        fetchType: FETCH

  - id: notify
    type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
    url: "{{ secret('SLACK_URL') }}"
    payload: |
      {
        "text": "Sales extract for {{ inputs.run_date }} complete",
        "attachments": [
          {
            "title": "Per-table row counts",
            "text": "{{ outputs.per_table | json_encode }}"
          }
        ]
      }

errors:
  - id: notify_failure
    type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
    url: "{{ secret('SLACK_URL') }}"
    payload: |
      {
        "text": "❌ Sales extract FAILED — run_date={{ inputs.run_date }} — {{ execution.id }}"
      }

triggers:
  - id: nightly
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 3 * * *"
    timezone: UTC

Step-by-step trace.

Layer	Component	Reasoning
Table discovery	list_tables (FETCH)	small result set; safe to load into memory
Parallel iteration	EachParallel, concurrent=4	bounded fan-out; protects source Postgres
Per-table extract	extract (STORE)	Parquet file per table; no memory pressure
Upload	S3 Upload from URI	zero-copy between tasks
Row-count for reporting	count_rows (FETCH)	small; feeds Slack payload
Success notify	Slack Webhook	payload built from outputs.per_table
Failure notify	errors: Slack Webhook	declarative on-failure handler

After the flow ships, an operator's manual backfill for a specific date is kestra flow trigger prod.warehouse sales_multi_table_extract --input run_date=2026-07-15; the flow runs the same code path as the scheduled 03:00 UTC execution, just with the operator-supplied date. Failures automatically ping Slack; successes report per-table row counts.

Output:

Task	Runs	Notes
list_tables	1	returns N table names
per_table (EachParallel)	1 (aggregates N iterations)	concurrent=4 caps in-flight
extract	N (one per table)	STORE writes Parquet per table
upload	N	S3 key includes table name + date
count_rows	N	feeds Slack payload
notify	1	posted once with all per-table counts

Why this works — concept by concept:

• Typed DATE input with default — the flow works identically for scheduled and manual runs. Scheduled runs feed run_date from the trigger; manual backfills pass it via CLI or UI. One code path, two invocation modes.
• EachParallel with concurrent cap — bounded fan-out. Without the cap, a 500-table schema would launch 500 concurrent Postgres queries and crush the source. concurrent: 4 is the throttle knob every senior flow uses.
• STORE fetchType for large results — extract writes a Parquet file to Kestra's file store rather than holding rows in memory. Downstream upload consumes the URI directly; zero copies through Kestra memory.
• Errors block for failure notifications — declarative on-failure handler that runs only on flow failure. Removes the callback-based error handling of Airflow (on_failure_callback, on_retry_callback) in favour of a top-level YAML block.
• Cost — one Postgres round-trip for discovery, N parallel extracts (capped at 4), N uploads, N count queries, one Slack post. Compared to the equivalent Airflow flow (dynamic task mapping + BranchPythonOperator + provider imports + XCom-based fan-in), this is dramatically less code and fewer moving parts. Net O(N/concurrent) end-to-end latency for N tables — the concurrency cap is the only tunable knob.

SQL
Topic — sql
SQL problems on multi-table extract patterns

Practice →

Design Topic — design Design problems on declarative workflow engines

Practice →

---

3. Plugin ecosystem + integrations

kestra plugin are drop-in JAR files — 600+ first-class integrations across warehouses, lakes, transforms, and streams

The mental model in one line: a kestra plugin is a JVM JAR file dropped into /app/plugins/ at cluster boot that registers one or more typed task classes (io.kestra.plugin.gcp.bigquery.Query, io.kestra.plugin.dbt.cli.DbtCLI, io.kestra.plugin.aws.s3.Upload) plus optional trigger classes and condition classes, with each class annotated @Plugin and its properties documented via JSON schema for the UI editor; the plugin catalog covers 600+ integrations across warehouses (Snowflake, BigQuery, Redshift, Clickhouse), lakes (S3, GCS, Azure Blob, MinIO), transforms (dbt, Airbyte, Fivetran, Meltano), streams (Kafka, Pulsar, Google Pub/Sub, AWS SQS), and utilities (Slack, PagerDuty, Postgres, MongoDB, Elasticsearch), and the plugin model gives Kestra the same "one flow, many integrations" ergonomics Airflow gets from providers — but with JVM-level dependency isolation rather than pip-level conflict roulette.

The plugin taxonomy — five families that cover 90% of platforms.

• Warehouse plugins. Snowflake (io.kestra.plugin.jdbc.snowflake.Query), BigQuery (io.kestra.plugin.gcp.bigquery.Query), Redshift (io.kestra.plugin.jdbc.redshift.Query), Clickhouse, DuckDB, Databricks SQL, Presto/Trino, Postgres, MySQL, MariaDB, SQL Server, Oracle. Every warehouse worth mentioning has a Query task type that runs SQL and returns rows via fetchType: FETCH or a file via fetchType: STORE.
• Lake / storage plugins. S3 (io.kestra.plugin.aws.s3.Upload / Download / List), GCS (io.kestra.plugin.gcp.gcs.Upload / Download / List), Azure Blob (io.kestra.plugin.azure.storage.blob.*), MinIO, HDFS. Every cloud object store has a matching plugin with the same Upload / Download / List / Delete task shape.
• Transform plugins. dbt (io.kestra.plugin.dbt.cli.DbtCLI), Airbyte (io.kestra.plugin.airbyte.connections.Sync), Fivetran (io.kestra.plugin.fivetran.connectors.Trigger), Meltano, Great Expectations, Soda. These plugins wrap the CLI or REST API of the transform tool and expose Kestra-native inputs and outputs.
• Stream plugins. Kafka (io.kestra.plugin.kafka.Produce / Consume / Trigger), Pulsar, AWS SQS, Google Pub/Sub, RabbitMQ, MQTT, WebSocket. Both task types (for one-shot produce/consume) and trigger types (for Realtime streaming ingestion) are shipped in the same plugin.
• Utility plugins. Slack (io.kestra.plugin.notifications.slack.SlackIncomingWebhook), PagerDuty, Discord, Email, Telegram, MS Teams, GitHub, GitLab, Jira, ServiceNow, Vault, AWS Secrets Manager, GCP Secret Manager. The plumbing that turns a flow into a proper on-call-friendly operation.

Plugin installation — three pathways to load a JAR.

• The bundled image. kestra/kestra:latest ships with 100+ plugins pre-installed. For most warehouses / lakes / notifications, no extra install is needed. Check the plugin catalog for the bundled list.
• A custom Docker image. The 12-factor answer: FROM kestra/kestra:latest and copy your custom or third-party plugin JARs into /app/plugins/. Rebuild the image, redeploy the cluster. The plugin manifest is loaded at JVM boot; new plugins appear in the UI on next start.
• The plugin install command. For dev / interactive scenarios, kestra plugins install io.kestra.plugin:plugin-airbyte:LATEST downloads and installs a plugin at runtime — but the plugin is lost on container restart unless the image was baked with it.

Plugin discovery — from the UI and CLI.

• The UI plugin catalog. /plugins in the Kestra web UI lists every installed plugin with class name, group, version, and JSON schema for the properties. Clicking a plugin drills into its task types, their inputs, and their outputs. This is the "which parameters does this task accept" reference.
• The CLI. kestra plugins list prints the same catalog. kestra plugins schema io.kestra.plugin.gcp.bigquery.Query prints the JSON schema for a specific task type. Useful for scripting and CI validation.
• The autocomplete. The UI editor uses the schemas for live autocomplete — typing type: inside a tasks: list shows every installed task class; typing a property name shows every documented field. This is the concrete win of typed plugins over Airflow's introspection-based provider docs.

Writing a custom plugin — five lines of Java + Micronaut DI.

• The class. Java class annotated @Plugin (from io.kestra.core.models.annotations.Plugin) plus @Schema for documentation. Extends Task and implements run(RunContext).
• The build. Gradle build with kestra-processor annotation processor; produces a JAR with the plugin manifest baked in.
• The install. Drop the JAR into /app/plugins/, restart cluster. Plugin appears in /plugins UI.
• The invocation. From YAML, type: com.acme.plugin.mycustom.MyTask — same shape as any built-in plugin.

Common interview probes on plugins.

• "How do Kestra plugins work?" — required answer: drop-in JARs loaded at boot; each registers @Plugin-annotated task classes.
• "How is that different from Airflow providers?" — Airflow providers are pip packages; Kestra plugins are JARs. Dependency isolation is at the JVM classloader vs pip package level.
• "How do you write a custom plugin?" — Java class + @Plugin annotation + Gradle build + JAR drop.
• "How do you install a third-party plugin?" — bake into custom Docker image, or kestra plugins install for dev.

Worked example — dbt run + BigQuery load + S3 archive in one flow

Detailed explanation. A canonical polyglot flow that exercises three plugins in sequence: dbt runs SQL transforms against BigQuery, the resulting fact table is exported to a GCS bucket, and the export is copied to S3 for cross-cloud disaster recovery. Every task is a first-class plugin; the flow authoring is pure YAML.

• Task 1. dbt run for a specific model tag.
• Task 2. BigQuery Query that exports the fact table to GCS.
• Task 3. GCS Download → S3 Upload for cross-cloud replication.
• Task 4. Slack notification with run summary.

Question. Write the polyglot flow that composes dbt, BigQuery, GCS, S3, and Slack plugins in one YAML file.

Input.

Component	Plugin	Purpose
dbt run	io.kestra.plugin.dbt.cli.DbtCLI	build fact_orders model
BigQuery export	io.kestra.plugin.gcp.bigquery.ExtractToGcs	dump table to GCS Parquet
GCS download	io.kestra.plugin.gcp.gcs.Download	pull GCS file into Kestra store
S3 upload	io.kestra.plugin.aws.s3.Upload	mirror to S3 for DR
Slack notify	io.kestra.plugin.notifications.slack.SlackIncomingWebhook	run summary

Code.

id: nightly_fact_orders_build
namespace: prod.warehouse
description: "dbt run → BigQuery export → cross-cloud S3 mirror"

labels:
  team: warehouse
  tier: production

inputs:
  - id: run_date
    type: DATE
    defaults: "{{ execution.startDate | date('yyyy-MM-dd') }}"

tasks:
  - id: dbt_build
    type: io.kestra.plugin.dbt.cli.DbtCLI
    projectDir: /app/dbt-project
    commands:
      - dbt run --select tag:fact_orders --vars '{"run_date": "{{ inputs.run_date }}"}'
      - dbt test --select tag:fact_orders
    profiles: |
      warehouse:
        target: prod
        outputs:
          prod:
            type: bigquery
            method: service-account-json
            keyfile_json: "{{ secret('GCP_SA_JSON') | fromJson }}"
            project: acme-analytics-prod
            dataset: analytics
            threads: 8
            timeout_seconds: 300
            location: US

  - id: export_to_gcs
    type: io.kestra.plugin.gcp.bigquery.ExtractToGcs
    serviceAccount: "{{ secret('GCP_SA_JSON') }}"
    projectId: acme-analytics-prod
    sourceTable: analytics.fact_orders
    destinationUris:
      - "gs://acme-analytics-exports/fact_orders/dt={{ inputs.run_date }}/data-*.parquet"
    format: PARQUET
    compression: SNAPPY

  - id: download_from_gcs
    type: io.kestra.plugin.gcp.gcs.Download
    serviceAccount: "{{ secret('GCP_SA_JSON') }}"
    from: "gs://acme-analytics-exports/fact_orders/dt={{ inputs.run_date }}/data-000000000000.parquet"

  - id: upload_to_s3
    type: io.kestra.plugin.aws.s3.Upload
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
    region: us-east-1
    bucket: acme-dr-mirror
    key: "fact_orders/dt={{ inputs.run_date }}/data.parquet"
    from: "{{ outputs.download_from_gcs.uri }}"

  - id: notify_success
    type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
    url: "{{ secret('SLACK_URL') }}"
    payload: |
      {
        "text": "Nightly fact_orders build complete for {{ inputs.run_date }}",
        "attachments": [{
          "title": "dbt build",
          "text": "{{ outputs.dbt_build.stdout | truncate(500) }}"
        }, {
          "title": "S3 mirror",
          "text": "s3://acme-dr-mirror/fact_orders/dt={{ inputs.run_date }}/data.parquet"
        }]
      }

triggers:
  - id: nightly
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 4 * * *"
    timezone: UTC

errors:
  - id: notify_failure
    type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
    url: "{{ secret('SLACK_URL') }}"
    payload: |
      {
        "text": "❌ fact_orders build FAILED — run_date={{ inputs.run_date }} — {{ execution.id }}"
      }

Step-by-step explanation.

1. The dbt_build task uses the dbt plugin (io.kestra.plugin.dbt.cli.DbtCLI). The commands: list runs dbt run and dbt test; the profiles: block is generated dynamically per execution so the same flow works across environments (dev / staging / prod). No profiles.yml file on disk; Kestra materialises it in the container's working directory for the task's lifetime.
2. The export_to_gcs task uses the BigQuery plugin's ExtractToGcs task type — a specialised task that runs EXPORT DATA under the hood, exports the fact table to GCS as Parquet, and emits the destination URIs as outputs. The plugin knows about BigQuery's native export mechanics; you never write the boilerplate.
3. The download_from_gcs + upload_to_s3 sequence is the cross-cloud copy. The GCS plugin downloads the file into Kestra's internal file store (outputs.download_from_gcs.uri); the S3 plugin consumes that URI as its from: field. Two plugins, one flow, zero manual data movement.
4. The notify_success task uses the Slack plugin. The payload is a JSON string built via Pebble; {{ outputs.dbt_build.stdout | truncate(500) }} grabs the dbt CLI stdout (up to 500 chars) as an attachment. The Pebble truncate filter keeps the Slack payload small.
5. The errors: block runs only on failure. The failure notification uses the same Slack plugin as the success one; the only difference is the message text. No callback hell; the failure handler is a top-level YAML block.

Output.

Task	Output	Downstream reference
dbt_build	stdout (log), exitCode	stdout in Slack payload
export_to_gcs	destinationUris (list of GCS URIs)	(side-effect; not consumed)
download_from_gcs	uri (Kestra file store URI)	consumed by upload_to_s3
upload_to_s3	(side-effect only)	s3 URL in Slack payload
notify_success	(side-effect only)	Slack post

Rule of thumb. For any polyglot flow, name the plugin class explicitly in every type: field — do not rely on shorthand aliases. The full class name (io.kestra.plugin.dbt.cli.DbtCLI) is self-documenting and survives plugin renames. Copy-paste from the UI plugin catalog when in doubt.

Worked example — Kafka Realtime trigger + Postgres upsert

Detailed explanation. Kestra's Realtime trigger turns a plugin's consumer into a flow spawner: one message → one flow execution. For a Kafka topic feeding a Postgres upsert, the entire "consume, transform, load" pattern is a single flow. Walk through the design.

• Trigger. Kafka Realtime trigger consuming a topic; each message spawns a flow execution.
• Task 1. Transform the incoming message with a Python script.
• Task 2. Upsert the transformed row into Postgres.
• Task 3. Send an ack back to another Kafka topic (audit trail).

Question. Write the streaming flow that reacts to each Kafka message with a Python transform + Postgres upsert + Kafka ack.

Input.

Component	Value
Source topic	events.orders
Ack topic	events.orders.ack
Postgres table	analytics.orders_stream
Consumer group	kestra-orders-ingest

Code.

id: orders_stream_ingest
namespace: prod.streaming
description: "Realtime Kafka → Postgres upsert with ack"

tasks:
  - id: transform
    type: io.kestra.plugin.scripts.python.Script
    script: |
      import json
      from kestra import Kestra

      # The Realtime trigger exposes the message under {{ trigger.value }}
      msg = json.loads({{ trigger.value | json_encode }})

      # Normalise + enrich
      row = {
          "order_id":     msg["orderId"],
          "customer_id":  msg["customer"]["id"],
          "total_cents":  int(msg["total"] * 100),
          "status":       msg["status"],
          "occurred_at":  msg["timestamp"],
      }
      Kestra.outputs({"row": row, "order_id": row["order_id"]})

  - id: upsert
    type: io.kestra.plugin.jdbc.postgresql.Query
    url: "jdbc:postgresql://analytics-db.internal:5432/analytics"
    username: "{{ secret('PG_ANALYTICS_USER') }}"
    password: "{{ secret('PG_ANALYTICS_PASSWORD') }}"
    sql: |
      INSERT INTO analytics.orders_stream (order_id, customer_id, total_cents, status, occurred_at)
      VALUES ({{ outputs.transform.vars.row.order_id }},
              {{ outputs.transform.vars.row.customer_id }},
              {{ outputs.transform.vars.row.total_cents }},
              '{{ outputs.transform.vars.row.status }}',
              '{{ outputs.transform.vars.row.occurred_at }}')
      ON CONFLICT (order_id) DO UPDATE
        SET customer_id = EXCLUDED.customer_id,
            total_cents = EXCLUDED.total_cents,
            status      = EXCLUDED.status,
            occurred_at = EXCLUDED.occurred_at

  - id: ack
    type: io.kestra.plugin.kafka.Produce
    properties:
      bootstrap.servers: kafka:9092
      key.serializer: org.apache.kafka.common.serialization.StringSerializer
      value.serializer: org.apache.kafka.common.serialization.StringSerializer
    topic: events.orders.ack
    from:
      - key: "{{ outputs.transform.vars.order_id }}"
        value: |
          { "order_id": {{ outputs.transform.vars.order_id }}, "ingested_at": "{{ execution.startDate }}" }

triggers:
  - id: kafka_source
    type: io.kestra.plugin.kafka.RealtimeTrigger
    properties:
      bootstrap.servers: kafka:9092
      group.id: kestra-orders-ingest
      key.deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value.deserializer: org.apache.kafka.common.serialization.StringDeserializer
    topic: events.orders
    serdeProperties:
      valueDeserializer: JSON

Step-by-step explanation.

1. The RealtimeTrigger sits under triggers: like any Schedule trigger, but its semantics are different: it spawns one flow execution per Kafka message rather than one per cron tick. The message key is exposed as {{ trigger.key }} and the value as {{ trigger.value }} — Pebble-accessible in every downstream task.
2. The transform task uses the Python plugin's Script type. Inside, {{ trigger.value | json_encode }} embeds the incoming Kafka value as a JSON string that Python parses; the transform normalises + enriches and emits a row and a order_id via Kestra.outputs.
3. The upsert task uses the Postgres plugin's Query type. The SQL is a straight INSERT ... ON CONFLICT DO UPDATE — the idempotent upsert pattern. All values are bound via Pebble from the upstream outputs.transform.vars.row; no manual JDBC parameter binding.
4. The ack task uses the Kafka plugin's Produce type. It publishes a small ack message to events.orders.ack with the order ID as the key (for downstream partition-based consumers) and a JSON payload including the ingest timestamp. This is the audit-trail primitive.
5. Because RealtimeTrigger spawns one execution per message, the Kestra executor becomes a lightweight streaming processor. For small-to-medium throughput (thousands of messages per second per topic), this is a legitimate alternative to a dedicated Flink or Spark Streaming job — you get the same "consume + transform + write" loop with far less infrastructure.

Output.

Task	Output	Notes
transform	vars.row (dict), vars.order_id (int)	flows through Pebble into upsert
upsert	(SQL side-effect only)	Postgres row upserted
ack	(Kafka side-effect only)	ack message published

Rule of thumb. For any Kestra streaming flow, use RealtimeTrigger with the appropriate stream plugin (Kafka, Pulsar, Pub/Sub) and treat every execution as a single-message flow. If throughput exceeds ~5000 messages/sec, switch to a dedicated stream processor (Flink, Spark) — Kestra's Realtime trigger is the right tool for the "moderate throughput, plugin-rich" band.

Worked example — writing a custom Java plugin

Detailed explanation. When no plugin covers a required integration, write a custom one. The plugin API is a small Java class annotated @Plugin plus a @Schema for documentation; the Gradle build produces a JAR that drops into /app/plugins/. Walk through a minimal custom plugin that hashes a string and returns the digest — the plugin author's "hello world."

• Class. HashDigest extends Task, implements run(RunContext).
• Properties. One input: the string to hash. One output: the digest.
• Build. Gradle with kestra-processor annotation processor.
• Install. JAR into /app/plugins/, restart cluster.

Question. Write the minimal custom plugin that computes a SHA-256 digest of an input string, and show how to invoke it from a YAML flow.

Input.

Component	Value
Class	com.acme.plugin.hash.HashDigest
Input property	text: String
Output field	digest: String (hex)
Task type in YAML	com.acme.plugin.hash.HashDigest

Code.

// src/main/java/com/acme/plugin/hash/HashDigest.java
package com.acme.plugin.hash;

import io.kestra.core.models.annotations.Example;
import io.kestra.core.models.annotations.Plugin;
import io.kestra.core.models.annotations.PluginProperty;
import io.kestra.core.models.tasks.RunnableTask;
import io.kestra.core.models.tasks.Task;
import io.kestra.core.runners.RunContext;
import io.swagger.v3.oas.annotations.media.Schema;
import lombok.*;
import lombok.experimental.SuperBuilder;

import java.security.MessageDigest;

@SuperBuilder
@ToString
@EqualsAndHashCode
@Getter
@NoArgsConstructor
@Schema(title = "SHA-256 hash of an input string.")
@Plugin(
    examples = {
        @Example(
            code = {
                "text: hello world"
            }
        )
    }
)
public class HashDigest extends Task implements RunnableTask<HashDigest.Output> {

    @Schema(title = "Text to hash")
    @PluginProperty(dynamic = true)
    private String text;

    @Override
    public Output run(RunContext runContext) throws Exception {
        String rendered = runContext.render(this.text);
        MessageDigest md = MessageDigest.getInstance("SHA-256");
        byte[] hash = md.digest(rendered.getBytes("UTF-8"));

        StringBuilder hex = new StringBuilder();
        for (byte b : hash) hex.append(String.format("%02x", b));

        return Output.builder()
            .digest(hex.toString())
            .build();
    }

    @Builder
    @Getter
    public static class Output implements io.kestra.core.models.tasks.Output {
        @Schema(title = "SHA-256 hex digest")
        private final String digest;
    }
}

// build.gradle
plugins {
    id 'java-library'
    id 'com.gradleup.shadow' version '8.3.0'
}

repositories { mavenCentral() }

dependencies {
    compileOnly 'io.kestra:core:0.20.0'
    annotationProcessor 'io.kestra:processor:0.20.0'
    compileOnly 'org.projectlombok:lombok:1.18.30'
    annotationProcessor 'org.projectlombok:lombok:1.18.30'
}

jar {
    manifest {
        attributes 'X-Kestra-Name': 'plugin-hash',
                   'X-Kestra-Group': 'com.acme.plugin.hash',
                   'X-Kestra-Title': 'Hash plugin',
                   'X-Kestra-Description': 'SHA-256 hashing task'
    }
}

# invoked from a flow just like any built-in plugin
id: hash_demo
namespace: dev.demo

tasks:
  - id: hash_it
    type: com.acme.plugin.hash.HashDigest
    text: "{{ inputs.payload }}"

  - id: log_digest
    type: io.kestra.plugin.core.log.Log
    message: "SHA-256({{ inputs.payload }}) = {{ outputs.hash_it.digest }}"

inputs:
  - id: payload
    type: STRING
    defaults: "hello world"

Step-by-step explanation.

1. The HashDigest class extends Task (Kestra's base task type) and implements RunnableTask<Output> (parameterised by the output shape). The @Schema annotation on the class provides the description shown in the UI plugin catalog; @Plugin with an examples: block populates the "Copy example" button in the UI.
2. The text property is annotated @PluginProperty(dynamic = true) — this tells the Kestra runtime that Pebble templating should be applied to the value before the task runs. Inside run(), runContext.render(this.text) evaluates the template; without dynamic = true, the value passes through verbatim.
3. The run() method receives a RunContext (Kestra's per-execution context) and returns an Output object. The digest computation is straight Java MessageDigest; the output builder pattern (from Lombok's @Builder) is Kestra's convention.
4. The Gradle build uses kestra:processor annotation processor to generate the plugin manifest (a JSON file baked into the JAR's META-INF/). The X-Kestra-* manifest attributes register the plugin in the UI catalog. shadow builds a fat JAR with dependencies embedded; drop the resulting JAR into /app/plugins/.
5. From YAML, invoking the plugin looks identical to any built-in: type: com.acme.plugin.hash.HashDigest followed by the properties. The UI editor autocompletes the properties from the JSON schema generated by the annotation processor. This is what "plugin as first-class citizen" means in practice — custom plugins are indistinguishable from built-ins.

Output.

Layer	Artifact	Purpose
Source	HashDigest.java	plugin class
Build	plugin-hash-1.0.jar	shadow JAR with deps
Install	/app/plugins/plugin-hash-1.0.jar	JVM classloader picks it up on boot
UI catalog	/plugins/com.acme.plugin.hash.HashDigest	discoverable + autocompletable
YAML usage	type: com.acme.plugin.hash.HashDigest	just like any built-in

Rule of thumb. For any custom plugin, keep the class small (one task type per class), always mark string properties with @PluginProperty(dynamic = true) so Pebble works, and bake the plugin into a versioned custom Docker image (FROM kestra/kestra:0.20 && COPY plugin-hash-1.0.jar /app/plugins/) — that way the plugin lifecycle is git-controlled, not runtime-installed.

Senior interview question on the plugin ecosystem

A senior interviewer might ask: "Compare the Kestra plugin model to Airflow's provider system. Walk through how a new integration reaches production in each, the dependency-conflict story, the on-call cost, and when you'd write a custom plugin in each system."

Solution Using JAR-based plugins + Docker-baked deployment + Java custom-plugin path

# 1. Pinned custom Docker image — bakes plugins into the artifact
# Dockerfile
# ---------------------------------
# FROM kestra/kestra:0.20.0
# COPY custom-plugins/*.jar /app/plugins/
# ---------------------------------

# 2. Custom-plugins directory contents (baked into the image)
# custom-plugins/
#   plugin-hash-1.2.jar         # com.acme.plugin.hash.HashDigest
#   plugin-mongo-1.1.jar        # com.acme.plugin.mongo.MongoDump
#   plugin-splunk-0.9.jar       # com.acme.plugin.splunk.SplunkForward

# 3. Reference flow using both built-in and custom plugins
id: nightly_pii_hash_and_archive
namespace: prod.privacy
description: "Extract PII, hash sensitive columns via custom plugin, archive to S3"

tasks:
  - id: extract_users
    type: io.kestra.plugin.jdbc.postgresql.Query
    url: "jdbc:postgresql://db-primary.internal:5432/production"
    username: "{{ secret('PG_USER') }}"
    password: "{{ secret('PG_PASSWORD') }}"
    sql: "SELECT id, email, phone FROM public.users"
    fetchType: FETCH

  - id: hash_each_email
    type: io.kestra.plugin.core.flow.EachParallel
    concurrent: 16
    value: "{{ outputs.extract_users.rows | map(attribute='email') | list }}"
    tasks:
      - id: hash_email
        type: com.acme.plugin.hash.HashDigest
        text: "{{ taskrun.value }}"

  - id: upload_manifest
    type: io.kestra.plugin.aws.s3.Upload
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
    region: us-east-1
    bucket: acme-privacy-archive
    key: "hashed-users/dt={{ execution.startDate | date('yyyy-MM-dd') }}/digest.json"
    from: "{{ outputs.hash_each_email | json_encode }}"

  - id: notify
    type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
    url: "{{ secret('SLACK_URL') }}"
    payload: |
      {"text": "PII hash archive complete — {{ outputs.extract_users.rows | length }} users hashed."}

triggers:
  - id: nightly
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 5 * * *"

Step-by-step trace.

Layer	Concern	Answer
Plugin isolation	JVM classloader per plugin	no pip / dependency conflicts across plugins
Plugin discovery	UI /plugins + CLI kestra plugins list	schema-driven autocomplete
Custom plugin	Java @Plugin class + Gradle	JAR drop into /app/plugins/
Deployment	custom Docker image FROM kestra/kestra:X	reproducible; git-managed
On-call cost	plugin upgrades are image rebuilds	one image, one restart
Airflow contrast	pip package + provider docs	pip conflict on cross-provider version pins

After the flow ships, the platform team owns one Dockerfile listing every plugin JAR (built-ins are already baked into kestra/kestra:0.20.0; only custom ones need COPY). Plugin upgrades are Dockerfile edits + image rebuild + rolling restart — the entire cluster's plugin state is a single git-tracked artifact.

Output:

Metric	Kestra plugin model	Airflow provider model
Install unit	JAR file	pip package
Isolation	JVM classloader	Python process (shared site-packages)
Conflict story	zero (classloader per plugin)	pip install version-pin fights
Discovery	UI /plugins + JSON schema	Airflow provider docs + docstrings
Custom-plugin lang	Java (or JVM lang)	Python
Deployment	Docker image with JARs	Docker image with pip packages
Upgrade blast radius	one JAR replaced	one package upgraded, potential deps rebuild

Why this works — concept by concept:

• JAR classloader isolation — every plugin JAR is loaded in its own classloader; two plugins can depend on incompatible library versions without a conflict. Airflow's pip-based providers share the same site-packages, so a version pin from one provider constrains the whole install.
• Docker-baked deployment — the custom Dockerfile becomes the entire plugin manifest. Deploying a new plugin is git commit + docker build + rollout. No runtime plugin install; no drift between staging and prod.
• Schema-driven UI autocomplete — every plugin exposes a JSON schema for its properties; the UI editor uses the schema for autocomplete and validation. The @PluginProperty(dynamic = true) annotation tells the runtime to render Pebble; the UI shows a helpful "supports templating" hint.
• Custom plugins are first-class — from YAML, type: com.acme.plugin.hash.HashDigest is indistinguishable from type: io.kestra.plugin.core.log.Log. The plugin catalog treats them identically; the UI editor autocompletes them identically. This is what "no second-class citizens" looks like in a plugin architecture.
• Cost — one custom Docker image per Kestra release, one Java project per custom plugin, zero pip-conflict investigations. Compared to Airflow's periodic "why does upgrading apache-airflow-providers-google break apache-airflow-providers-amazon" incidents, the JAR model buys a step-change reduction in on-call cost. Net O(plugins) build time versus O(plugins²) pip-conflict investigation time.

Design
Topic — design
Design problems on plugin architectures

Practice →

SQL Topic — sql SQL problems on multi-source orchestration

Practice →

---

4. Executors, workers and queue architecture

The executor / worker split on Kafka + Postgres is the reason Kestra scales cleanly — no Celery, no K8s religion required

The mental model in one line: a production Kestra cluster is five components — executor (the flow scheduler), worker (the task runner), webserver (the UI + API), scheduler (the trigger evaluator), and indexer (the search backfiller) — connected by Kafka topics for the message bus, backed by Postgres for durable state, and optionally by Elasticsearch for the UI's search / history index; the executor / worker split lets you scale scheduling and execution independently, and worker groups (worker.group=<tag>) let you route different task types to different worker fleets — GPU workers for ML tasks, K8s-launcher workers for container tasks, standard workers for everything else. This architecture is a step function over Airflow's Celery / Kubernetes / Local executor debate — one uniform mental model, one set of scaling knobs.

The five cluster components — what each one owns.

• Executor. The flow orchestrator — reads flow definitions from Postgres, tracks execution state (which task is next, which succeeded/failed), publishes task-run requests to a Kafka topic, and consumes task-run results. One executor per cluster is enough for tens of thousands of flows; run two or three for HA. Executors are stateless — state lives in Kafka + Postgres.
• Worker. The task runner — subscribes to the worker-tasks Kafka topic, pulls task-run requests, executes them (runs the Python script, issues the JDBC query, uploads the S3 file), publishes task-run results back to Kafka. Workers are stateless and horizontally scalable; scale to the concurrency you need.
• Webserver. The UI + REST API — serves the web console, accepts flow CRUD, exposes the execution history, streams live logs. Read-mostly; scale for user concurrency (usually one or two is enough).
• Scheduler. The trigger evaluator — reads flow triggers from Postgres, evaluates cron schedules and Realtime consumers, publishes execution start requests to Kafka. Also one per cluster is typical; run two for HA.
• Indexer. The search backfiller — reads execution results from Kafka, writes them into Elasticsearch (or Postgres, in non-EE editions) for the UI's search and history views. Scale for indexing throughput on high-volume clusters.

The backend stack — three durable stores.

• Postgres (state store). Holds flow definitions, execution metadata, secret store, and (in non-EE editions) the queue itself. This is the durable source of truth; back it up.
• Kafka (message bus). Carries the task-run and execution messages between executor, worker, scheduler, and indexer. In smaller deployments, Postgres can host the queue too, but Kafka is preferred for high-volume clusters (>1000 executions/hour).
• Elasticsearch (search index). Backs the UI's execution search and history views. Optional; can be replaced by Postgres full-text search in smaller deployments.

The three deployment modes — dev, single-node prod, distributed prod.

• Standalone dev. kestra server standalone runs all five components in one JVM, with an embedded H2 database and in-memory queue. Zero config; the fastest way to try Kestra on a laptop.
• Single-node production. kestra server standalone --config /etc/kestra/config.yml with an external Postgres. Still one JVM, but state survives restarts. Fine for small production teams (<50 flows, <1000 executions/day).
• Distributed production. Executor, worker, webserver, scheduler, indexer each in its own JVM (or K8s pod), sharing Postgres + Kafka. Scale each component independently. Standard Helm chart deployment; the reference architecture for anything at scale.

Worker groups — routing tasks to specialised workers.

• The idea. Every worker registers with a worker.group tag (e.g. main, gpu, k8s-launcher, on-prem-vpc). Tasks can specify workerGroup: to be routed to a specific group.
• The mechanism. Each worker group subscribes to its own Kafka topic; tasks tagged with a group are published only to that topic; workers only consume from their assigned topic.
• The use cases. Route ML-training tasks to GPU workers; route Snowflake queries to workers with a private-VPC network path; route container tasks to workers colocated with the container runtime.
• The scaling knob. Each group scales independently. GPU workers can be 2 replicas while standard workers are 20; you pay for GPU capacity only in proportion to GPU workload.

Scaling knobs — where the tuning happens.

• Worker concurrency. kestra.tasks.tmp-dir + JVM heap + the plugin task types determine per-worker throughput. A modest 4 vCPU / 8 GB worker handles ~50 concurrent tasks for typical SQL / HTTP / plugin workloads.
• Kafka partitions. The worker-tasks and execution topics should have partition counts >= worker replica count for maximum parallelism. Increase partitions before increasing replicas.
• Postgres connection pool. The executor + webserver + scheduler + indexer all pool connections; size the pool at ~2× the worker count.
• Executor thread pool. For high-throughput clusters, bump kestra.executor.thread-count to match Kafka partition count.

Common interview probes on the architecture.

• "What are the components of a Kestra cluster?" — required answer: executor, worker, webserver, scheduler, indexer.
• "How does Kestra scale workers?" — Kafka-based queue, horizontal workers, worker groups for specialisation.
• "How is Kestra state managed?" — Postgres for definitions + metadata; Kafka for the queue; Elasticsearch (optional) for search.
• "What if the executor dies?" — run two executors; Kafka rebalances; in-flight work resumes from the last Kafka offset committed.

Worked example — standalone Docker Compose for dev

Detailed explanation. The fastest way to try Kestra is kestra server standalone running against an embedded H2 database. Great for learning the YAML syntax; not durable. The next step is a docker-compose.yml with one Kestra container + one Postgres container — durable state, single-node, still simple. Walk through the compose file.

• Compose services. Postgres + Kestra standalone.
• State. Postgres volume mounted; Kestra file store on a persistent volume.
• Ports. UI on 8080.

Question. Write the docker-compose.yml for a durable single-node Kestra dev environment.

Input.

Component	Purpose
postgres	state + queue backend
kestra	standalone server (all 5 components in one JVM)
volumes	postgres-data (DB) + kestra-storage (file store)
port	8080 → UI

Code.

# docker-compose.yml — single-node Kestra dev
version: "3.8"

services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: kestra
      POSTGRES_PASSWORD: kestra
      POSTGRES_DB: kestra
    volumes:
      - postgres-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U kestra"]
      interval: 10s
      timeout: 5s
      retries: 5

  kestra:
    image: kestra/kestra:latest
    command: server standalone
    user: root
    environment:
      KESTRA_CONFIGURATION: |
        datasources:
          postgres:
            url: jdbc:postgresql://postgres:5432/kestra
            driverClassName: org.postgresql.Driver
            username: kestra
            password: kestra
        kestra:
          server:
            basic-auth:
              enabled: false
          repository:
            type: postgres
          storage:
            type: local
            local:
              base-path: /app/storage
          queue:
            type: postgres
          tasks:
            tmp-dir:
              path: /tmp/kestra-wd/tmp
    ports:
      - "8080:8080"
    depends_on:
      postgres:
        condition: service_healthy
    volumes:
      - kestra-storage:/app/storage
      - /var/run/docker.sock:/var/run/docker.sock

volumes:
  postgres-data:
  kestra-storage:

Step-by-step explanation.

1. The postgres service uses Postgres 16 with a healthcheck. Kestra waits for Postgres to be healthy before starting (via depends_on: condition: service_healthy) — this avoids the "Kestra crashed on boot because Postgres wasn't ready" race.
2. The kestra service uses the latest Kestra image and starts in standalone mode. The KESTRA_CONFIGURATION env var (Kestra reads YAML from this env var) points at Postgres for the repository (flow definitions), the queue (execution messages), and uses a local file store at /app/storage.
3. The basic-auth.enabled: false setting disables the login prompt for dev; in prod you'd enable it and either set static credentials or wire in OAuth2/OIDC.
4. The queue.type: postgres setting uses Postgres for the message queue — fine for single-node dev. For distributed prod you'd switch to queue.type: kafka and add a Kafka service.
5. The /var/run/docker.sock mount lets Kestra launch Docker containers for io.kestra.plugin.docker.* task types — the "run a container as a task" plugin. Skip this mount if you don't need Docker-backed tasks.

Output.

Layer	State after
docker compose up -d	postgres + kestra containers running
UI	http://localhost:8080 (no login)
Flow creation	UI editor or CLI (kestra flow create --file flow.yml)
State durability	flow definitions + execution history survive restarts

Rule of thumb. Use the compose file above for any dev or single-node Kestra deployment. It's the smallest configuration that survives a restart. Do not use the standalone --config variant with SQLite or H2 for anything you care about surviving overnight.

Worked example — distributed Helm chart for production

Detailed explanation. For production, split the components onto separate K8s deployments so each can scale independently. The official Helm chart does this out of the box: one deployment per component (executor, worker, webserver, scheduler, indexer), plus dependencies on Postgres, Kafka, and Elasticsearch. Walk through the values.yaml overrides.

• Components. Executor, worker (with worker groups), webserver, scheduler, indexer.
• Backends. Postgres (external RDS), Kafka (external MSK or Confluent), Elasticsearch (optional).
• Scaling. Worker replica count via Horizontal Pod Autoscaler.

Question. Write a Helm values.yaml override for a distributed prod Kestra cluster with two worker groups (main + gpu).

Input.

Component	Replicas	Notes
executor	2	HA active-active
worker (main group)	10	standard workload, CPU workers
worker (gpu group)	2	GPU nodes for ML plugins
webserver	2	UI + API
scheduler	2	HA trigger evaluator
indexer	2	Elasticsearch backfill

Code.

# values.yaml — distributed Kestra prod
image:
  repository: kestra/kestra
  tag: "0.20.0"

configuration:
  kestra:
    repository:
      type: postgres
    queue:
      type: kafka
    storage:
      type: s3
      s3:
        bucket: acme-kestra-storage
        region: us-east-1
        access-key: "${AWS_ACCESS_KEY_ID}"
        secret-key: "${AWS_SECRET_ACCESS_KEY}"
    server:
      basic-auth:
        enabled: true
        realm: kestra
        username: admin
        password: "${KESTRA_ADMIN_PASSWORD}"
    elasticsearch:
      client:
        http-hosts:
          - "https://es.acme.internal:9200"
  datasources:
    postgres:
      url: "jdbc:postgresql://kestra-db.internal:5432/kestra"
      username: kestra
      password: "${POSTGRES_PASSWORD}"
      driverClassName: org.postgresql.Driver
  kafka:
    client:
      properties:
        bootstrap.servers: "kafka-1:9092,kafka-2:9092,kafka-3:9092"
        security.protocol: SASL_SSL
        sasl.mechanism: SCRAM-SHA-512
        sasl.jaas.config: >
          org.apache.kafka.common.security.scram.ScramLoginModule required
          username="${KAFKA_USER}" password="${KAFKA_PASSWORD}";

# Component deployments
executor:
  enabled: true
  replicas: 2
  resources:
    requests: {cpu: "2", memory: "4Gi"}
    limits:   {cpu: "4", memory: "8Gi"}

worker:
  # Main group — 10 replicas, standard CPU
  - name: main
    replicas: 10
    configuration:
      kestra.worker.group: main
    resources:
      requests: {cpu: "2", memory: "4Gi"}
      limits:   {cpu: "4", memory: "8Gi"}

  # GPU group — 2 replicas, GPU-tainted nodes
  - name: gpu
    replicas: 2
    configuration:
      kestra.worker.group: gpu
    nodeSelector:
      accelerator: nvidia-tesla-t4
    tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
    resources:
      requests: {cpu: "4", memory: "16Gi", "nvidia.com/gpu": "1"}
      limits:   {cpu: "8", memory: "32Gi", "nvidia.com/gpu": "1"}

webserver:
  replicas: 2

scheduler:
  replicas: 2

indexer:
  replicas: 2

postgresql:
  enabled: false   # external RDS

kafka:
  enabled: false   # external MSK

elasticsearch:
  enabled: false   # external ES

# Flow that routes an ML task to the GPU worker group
id: train_recommender
namespace: prod.ml

tasks:
  - id: train
    type: io.kestra.plugin.scripts.python.Script
    workerGroup: gpu
    beforeCommands:
      - pip install torch scikit-learn
    script: |
      import torch
      # ... model training on the GPU worker ...

  - id: publish_model
    type: io.kestra.plugin.aws.s3.Upload
    workerGroup: main
    # ... upload model artifact to S3 (no GPU needed) ...

Step-by-step explanation.

1. The configuration: block feeds Kestra's YAML config to every component (executor, worker, webserver, scheduler, indexer). The queue.type: kafka setting routes execution messages through Kafka rather than Postgres — mandatory at scale.
2. The worker array declares two worker groups. main has 10 replicas on standard CPU nodes; gpu has 2 replicas on GPU-tainted nodes with nvidia.com/gpu: 1 resource requests. Each group has its own K8s deployment; each subscribes to its own Kafka topic (kestra-worker-tasks-main vs kestra-worker-tasks-gpu).
3. Setting configuration.kestra.worker.group per deployment tells that worker fleet which group to identify as. Workers only consume tasks tagged with their group; the executor routes tagged tasks to the matching group's topic.
4. The postgresql, kafka, elasticsearch sub-charts are disabled — we use external managed services (RDS Postgres, MSK Kafka, managed ES) in production. The connection info comes from the top-level datasources, kafka, and elasticsearch configuration blocks.
5. In the flow YAML, tasks specify workerGroup: gpu to be routed to the GPU workers. Untagged tasks default to the main group (or whatever group is configured as default). This lets one flow mix CPU and GPU tasks, with each landing on the right hardware.

Output.

K8s deployment	Replicas	Role
kestra-executor	2	schedules tasks
kestra-worker-main	10	standard CPU tasks
kestra-worker-gpu	2	GPU-accelerated ML tasks
kestra-webserver	2	UI + API
kestra-scheduler	2	trigger evaluator
kestra-indexer	2	ES backfill
RDS Postgres	(external)	state + repository
MSK Kafka	(external)	message queue
ES	(external)	search index

Rule of thumb. For any distributed Kestra deployment, always split executor + worker + webserver at minimum. Use worker groups when you have hardware-specialised workloads (GPU, on-prem VPC, K8s-launcher). Size the Kafka topics with partition count >= worker replica count; that's the throughput ceiling.

Worked example — scaling and monitoring the queue

Detailed explanation. A production Kestra cluster needs the same "watch the queues" discipline as any Kafka-backed system. Monitor Kafka consumer lag on the worker topics, Postgres connection pool utilisation, and per-task-type latency. Walk through the Prometheus metrics and the alerts every senior deployment ships.

• Metric families. Kestra JVM metrics, Kafka consumer lag, Postgres connection pool, per-task-type runtime.
• Alerts. Worker lag > N seconds, connection pool > 80%, failed tasks > baseline.
• Runbook. Scale workers on sustained lag; rotate connections on pool saturation; investigate task-family regression.

Question. Write the Prometheus alert rules for a Kestra prod cluster.

Input.

Metric	Alert threshold
Worker Kafka lag	> 60s for 5m
Postgres pool utilisation	> 80% for 10m
Failed executions	> 5% of last hour
Executor thread starvation	> 90% for 5m

Code.

# prometheus-alerts.yml
groups:
  - name: kestra
    interval: 30s
    rules:
      - alert: KestraWorkerLagHigh
        expr: |
          max(kafka_consumergroup_lag{consumergroup=~"kestra-worker-.*"}) > 1000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Kestra worker Kafka lag > 1000 msgs for 5m"
          runbook: "Scale kestra-worker-<group> deployment; investigate slow task types."

      - alert: KestraPgPoolExhausted
        expr: |
          kestra_datasource_hikari_pool_usage{pool="postgres"} > 0.8
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Kestra Postgres pool utilisation > 80% for 10m"
          runbook: "Bump hikari.max-pool-size; investigate long-held transactions."

      - alert: KestraExecutionsFailing
        expr: |
          rate(kestra_execution_end_count{state="FAILED"}[1h])
            /
          rate(kestra_execution_end_count[1h]) > 0.05
        for: 30m
        labels:
          severity: critical
        annotations:
          summary: "> 5% of Kestra executions failed in the last hour"
          runbook: "Query UI for failed flows in the top offending namespaces; investigate common cause."

      - alert: KestraExecutorStarved
        expr: |
          kestra_executor_thread_pool_active / kestra_executor_thread_pool_size > 0.9
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Kestra executor thread pool > 90% saturated"
          runbook: "Bump kestra.executor.thread-count; add executor replica."

# Kestra JMX + Micrometer exporter (already built-in)
# Expose port 9090; scrape with:
# - job_name: 'kestra'
#   static_configs:
#     - targets: ['kestra-executor:8080', 'kestra-worker:8080', 'kestra-webserver:8080']

Step-by-step explanation.

1. Worker Kafka lag is the primary "am I keeping up" signal. kafka_consumergroup_lag per consumer group tells you how many task-run messages are queued. Alerting at 1000 messages for 5 minutes catches sustained backpressure before it becomes a p95 latency incident.
2. Postgres pool utilisation is the "am I about to deadlock" signal. Kestra's HikariCP pool exposes hikari_pool_usage; sustained > 80% means a query is hoarding connections. Bumping the pool size is a quick fix; finding the hoarding query is the real fix.
3. Failed-execution rate is the "is my platform healthy" signal. A step-change from < 1% to > 5% is a regression. The alert fires slowly (30 min at > 5%) to avoid noise from transient failures; on fire, the runbook is "find the top offending namespaces and investigate."
4. Executor thread starvation is the scheduling-side backpressure signal. If the executor's thread pool is 90% busy, scheduling latency degrades — new tasks wait longer to be dispatched to workers. Bumping thread count is cheap; adding an executor replica is the horizontal scaling move.
5. Kestra ships JVM + Micrometer metrics out of the box on the /prometheus endpoint of every component. Scrape from every executor, worker, webserver, scheduler, indexer pod; label metrics by component so alerts can specify the failing role.

Output.

Alert	Severity	Action
KestraWorkerLagHigh	warning	scale worker deployment
KestraPgPoolExhausted	warning	bump pool size; investigate hoarding
KestraExecutionsFailing	critical	investigate failure top-offenders
KestraExecutorStarved	warning	scale executor / bump threads

Rule of thumb. Every Kestra prod cluster needs the four alerts above from day one. The metrics are already exposed; you just need to wire Prometheus and write the rules. A cluster without lag alerts is one traffic spike away from a 3 AM incident.

Senior interview question on the distributed architecture

A senior interviewer might ask: "Design a Kestra deployment for 500 flows / 10000 executions per day / mixed CPU + GPU workloads / multi-region HA. Cover the K8s deployment shape, worker-group routing, the Kafka + Postgres + Elasticsearch backends, the scaling knobs, and the on-call runbook when worker lag spikes to 30 seconds."

Solution Using distributed Helm + Kafka + RDS + worker groups + Prometheus alerts

# values.yaml — 500 flows / 10k exec-per-day / multi-region HA cluster
image:
  repository: kestra/kestra
  tag: "0.20.0"

configuration:
  kestra:
    repository:
      type: postgres
    queue:
      type: kafka
    storage:
      type: s3
      s3:
        bucket: acme-kestra-storage
        region: us-east-1
    elasticsearch:
      client:
        http-hosts:
          - "https://es-cluster.internal:9200"
  datasources:
    postgres:
      url: "jdbc:postgresql://kestra-db-rw.internal:5432/kestra"
      username: kestra
      password: "${POSTGRES_PASSWORD}"
      hikari:
        maximum-pool-size: 40
  kafka:
    client:
      properties:
        bootstrap.servers: "kafka-1:9092,kafka-2:9092,kafka-3:9092"
        security.protocol: SASL_SSL
        sasl.mechanism: SCRAM-SHA-512

executor:
  replicas: 3
  resources:
    requests: {cpu: "4", memory: "8Gi"}
    limits:   {cpu: "8", memory: "16Gi"}
  configuration:
    kestra.executor.thread-count: 32

worker:
  - name: main
    replicas: 20
    configuration:
      kestra.worker.group: main
    resources:
      requests: {cpu: "2", memory: "4Gi"}
      limits:   {cpu: "4", memory: "8Gi"}
    autoscaling:
      enabled: true
      minReplicas: 10
      maxReplicas: 50
      targetCPUUtilizationPercentage: 70

  - name: gpu
    replicas: 4
    configuration:
      kestra.worker.group: gpu
    nodeSelector:
      accelerator: nvidia-tesla-t4
    resources:
      requests: {"nvidia.com/gpu": "1"}
      limits:   {"nvidia.com/gpu": "1"}

  - name: on-prem-vpc
    replicas: 3
    configuration:
      kestra.worker.group: on-prem-vpc
    nodeSelector:
      network-zone: on-prem-vpc

webserver:
  replicas: 3
  ingress:
    enabled: true
    hosts:
      - host: kestra.acme.internal
        paths:
          - path: /
            pathType: Prefix

scheduler:
  replicas: 2

indexer:
  replicas: 3

# External services (managed)
postgresql: {enabled: false}
kafka:       {enabled: false}
elasticsearch: {enabled: false}

# Kafka topic layout (external MSK; do not create in-chart)
# kestra-executor-consumer:  8 partitions, replication 3
# kestra-worker-tasks-main:  16 partitions, replication 3   # matches 16 avg workers
# kestra-worker-tasks-gpu:   4 partitions, replication 3
# kestra-worker-tasks-on-prem-vpc: 4 partitions, replication 3
# kestra-execution-events:   8 partitions, replication 3
# kestra-flow-events:        4 partitions, replication 3

Step-by-step trace.

Concern	Config	Reasoning
Multi-region HA	3 executor + 2 scheduler + 3 webserver replicas across 3 AZs	any single-AZ failure survivable
Worker scaling	HPA on main (10-50); fixed on gpu / on-prem	main scales elastically; specialised stays fixed
Backend choice	RDS Postgres + MSK Kafka + managed ES	zero backend ops burden
Postgres pool	maximum-pool-size: 40	(executor + workers + webserver + indexer) × pool need
Kafka partitions	16 for main worker topic	matches average worker count
Storage	S3 for shared file store	executors + workers see the same URIs
Alerts	4 Prometheus rules	worker lag, pool, failure rate, executor starvation

After the rollout, the cluster handles 10k executions/day at ~30 concurrent tasks at peak. Multi-region HA is protected by Postgres RDS multi-AZ + MSK multi-AZ + K8s pods spread across zones. The GPU worker group runs ML training tasks; the on-prem-vpc group runs Snowflake queries against tables in a locked-down VPC. Worker lag averages < 2 seconds; the HPA scales main workers up during morning batch windows and back down overnight.

Output:

Component	Replicas	Owns
executor	3	flow scheduling
worker-main	10-50 (HPA)	standard CPU tasks
worker-gpu	4	ML training tasks
worker-on-prem-vpc	3	tasks needing VPC access
webserver	3	UI + API
scheduler	2	trigger evaluation
indexer	3	ES backfill
RDS Postgres	multi-AZ	flow + execution state
MSK Kafka	multi-AZ	message queue
managed ES	multi-AZ	search index

Why this works — concept by concept:

• Executor / worker split — scheduling (executor) and execution (worker) scale independently. A morning batch surge scales workers via HPA without touching the executor; a scheduling backlog gets more executor replicas without adding worker capacity. Each component has one job.
• Worker groups — the routing primitive that lets one cluster serve heterogeneous workloads. GPU workers only run GPU-tagged tasks; on-prem-vpc workers only run VPC-tagged tasks; standard workers handle everything else. No wasted GPU capacity on SQL queries.
• Kafka partitions ≥ worker count — Kafka's partition count caps the parallelism of consumers within a group. Setting main topic to 16 partitions matches the average worker count; if HPA scales to 50 workers momentarily, only 16 can consume in parallel until more partitions are added. Plan for peak, not average.
• External managed backends — RDS Postgres for state, MSK Kafka for the queue, managed ES for search. Zero DBA-on-call burden; zero Kafka-cluster-tuning nightmare. Kestra becomes stateless in the compute plane; state ops delegated to specialists.
• Cost — ~30 vCPU baseline (3+10+3+3+2+3 = 24 vCPU worker + 6 vCPU control plane), scaling to ~80 vCPU peak; RDS db.r6g.xlarge; MSK 3-broker cluster; managed ES 3-node small cluster. Compared to an equivalent Airflow deployment (Celery + Redis + Postgres + Airflow scheduler + Airflow webserver + 20 worker replicas), roughly the same footprint but with a cleaner scaling model. Net O(replicas) scaling per component rather than O(Celery-conductor-choreography). The eliminated cost is the "which executor mode do we run" ongoing debate.

Design
Topic — design
Design problems on distributed orchestrator architecture

Practice →

Streaming Topic — streaming Streaming problems on Kafka-backed queues

Practice →

---

5. Kestra vs Airflow / Dagster / Prefect + interview signals

kestra vs airflow is the interview question — pick per team, not per hype cycle

The mental model in one line: kestra vs airflow is the wrong question if you frame it as "which is better" — the correct framing is "which fits your team's authoring surface, plugin needs, and execution model best" — Kestra wins when the team is polyglot and non-engineers author schedules and the plugin ecosystem covers your integrations, Airflow wins when the team is Python-heavy and custom Python operators are the daily workload, Dagster wins when software-defined asset lineage is the first-class UX, Prefect wins when flows are dynamically constructed at runtime, and the mature senior answer names all four with their sweet spots rather than picking one as universally superior. Every senior data-engineering interviewer asks this question because the wrong answer signals bandwagon-following; the right answer signals architectural maturity.

The four orchestrators, side by side.

• Kestra. YAML-first authoring; UI-authoring-grade; JVM plugin JARs; Kafka + Postgres backend; Schedule / Webhook / Flow / Realtime triggers. Sweet spot: polyglot teams where non-engineers need to author schedules and the plugin catalog covers the integrations.
• Airflow. Python DAG authoring; UI is read-mostly; pip-based providers; Celery / K8s / Local executor; Schedule / trigger-rule / sensor-based waiting. Sweet spot: Python-heavy teams with deep custom operator needs and mature Python-ecosystem integrations.
• Dagster. Python authoring with software-defined-asset decorators; UI shows asset materialisation graph; pip packages; Postgres + Daemon backend; asset-centric scheduling. Sweet spot: teams where asset lineage and check-based data quality are first-class UX requirements.
• Prefect. Python authoring with task decorators; SaaS-first UI + optional self-hosted server; pip packages; Prefect Cloud or self-hosted backend; dynamic runtime graphs. Sweet spot: teams that build flow topology at runtime based on data or config.

The four axes — how the orchestrators differ.

• Authoring surface. Kestra = YAML; Airflow, Dagster, Prefect = Python. YAML is a step-change lower authoring bar for non-engineers; Python is a step-change higher expressiveness for engineers.
• UI depth. Kestra = authoring-grade (create, edit, deploy from UI); Dagster = rich asset-graph explorer; Prefect Cloud = SaaS-polished; Airflow = read-mostly (view state, trigger runs, browse logs). Kestra and Dagster tie on UI ambition; Airflow lags.
• Plugin ecosystem. Kestra = 600+ JAR plugins with JVM isolation; Airflow = ~90 provider packages, largest volume by task count; Dagster + Prefect = pip packages, smaller catalogs but Python-native.
• Execution model. Kestra = executor + worker split on Kafka + Postgres; Airflow = Celery / K8s / Local executor with scheduler + worker; Dagster = Daemon + Postgres; Prefect = agent-based (Prefect Cloud) or server-based (self-hosted).

When Kestra wins — the five scenarios.

• Polyglot task fleet. Python + SQL + JavaScript + shell + containers all in the same flow, with each task type as a first-class plugin. Airflow can run any language via BashOperator, but the flow logic itself is Python; Kestra removes that ceiling.
• Non-engineer authoring. Analytics engineers, BI analysts, and ops teams can author flows via the web editor without learning a Python API. YAML + UI is the lowest-friction authoring surface in the modern orchestrator space.
• Plugin coverage matches your stack. If Kestra's 600+ plugin catalog includes your warehouses, lakes, transforms, and notifiers, you gain the JVM-isolation benefit (no pip conflicts) for free. Check the plugin catalog before deciding.
• Kafka-native message-bus fit. Teams that already run Kafka get a natural fit for Kestra's queue backend. Same team, same operational muscle memory.
• Declarative governance. YAML flows are trivially git-diffable, PR-reviewable, and rollback-friendly. Every flow change is a git commit; every deploy is a kestra flow update --file flow.yml.

When Airflow wins — the five scenarios.

• Python-only team. Everyone on the platform writes Python; no analysts or BI folks need to author flows. Airflow's Python API is the natural fit.
• Deep custom operators. You have a big library of custom Python operators that would need to be rewritten as Java plugins to move to Kestra. Migration cost outweighs the switch benefit.
• Airflow-native community integrations. Some niche integrations (specific ML tools, older warehouses) have Airflow providers but not Kestra plugins.
• Sensor-based waiting. Airflow's Sensor primitive (poll-until-condition-met) is more mature than Kestra's Poll task type. For sensor-heavy workloads, Airflow is still the reference.
• Cross-DAG dependencies via TriggerDagRunOperator. Airflow's cross-DAG orchestration is battle-tested; Kestra's Flow-trigger is newer but rapidly closing the gap.

When Dagster wins.

• Asset lineage is a first-class requirement; you want the UI to be an asset graph, not a task graph.
• Data-quality checks live at the asset level (@asset_check decorators).
• The team already thinks in "software-defined assets" and wants tooling that leans into that vocabulary.

When Prefect wins.

• Flow topology depends on runtime data; the graph is built dynamically per execution.
• SaaS-first UX is preferred (Prefect Cloud is the polished option; self-hosted is second-class).
• Team already uses Prefect and has invested in the ecosystem.

Common interview probes on the comparison.

• "When would you pick Kestra over Airflow?" — required answer: polyglot team + non-engineer authoring + plugin catalog fit.
• "When would you pick Airflow over Kestra?" — required answer: Python-only team + deep custom operators + Airflow-only integrations.
• "What about Dagster and Prefect?" — Dagster for asset lineage; Prefect for dynamic flows.
• "What's the migration cost between them?" — Airflow → Kestra: ~1 engineer-month per 100 DAGs (mostly rewriting Python operators as YAML flows); Kestra → Airflow: comparable in the reverse direction.

Worked example — the migration case study from Airflow

Detailed explanation. A 200-DAG Airflow platform migrating to Kestra: the operator inventory, the migration path per DAG family, the parity harness, and the cutover order. This is the canonical migration story senior candidates walk in interviews.

• Inventory. 200 DAGs, ~1200 tasks, mix of PythonOperator (60%), BashOperator (15%), SQLOperators (15%), custom (10%).
• Migration path. PythonOperator → Kestra Python Script plugin (mechanical); BashOperator → Kestra Shell Script plugin; SQLOperators → Kestra JDBC plugins; custom → rewrite as YAML with existing plugins or new custom plugin.
• Parity harness. Run both orchestrators for 4 weeks; diff execution graphs and output rows.

Question. Design the Airflow → Kestra migration plan for a 200-DAG platform.

Input.

Airflow operator family	Count	Kestra target
PythonOperator	720	io.kestra.plugin.scripts.python.Script
BashOperator	180	io.kestra.plugin.scripts.shell.Commands
SnowflakeOperator	90	io.kestra.plugin.jdbc.snowflake.Query
PostgresOperator	60	io.kestra.plugin.jdbc.postgresql.Query
S3ToRedshiftOperator	30	io.kestra.plugin.aws.s3.Upload + jdbc.redshift.Query
custom	120	rewrite as YAML with existing plugins or new plugin

Code.

# Airflow DAG — before migration (representative sample)
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.providers.snowflake.operators.snowflake import SnowflakeOperator
from airflow.providers.amazon.aws.transfers.s3_to_redshift import S3ToRedshiftOperator
from datetime import datetime

def compute_daily_revenue(ds, **_):
    # ... business logic ...
    return {"revenue_cents": 1234500}

with DAG(
    "daily_revenue",
    start_date=datetime(2024, 1, 1),
    schedule="0 6 * * *",
    catchup=False,
) as dag:
    compute = PythonOperator(
        task_id="compute",
        python_callable=compute_daily_revenue,
    )

    load_snowflake = SnowflakeOperator(
        task_id="load_snowflake",
        snowflake_conn_id="snowflake_prod",
        sql="""
            INSERT INTO analytics.daily_revenue(dt, revenue_cents)
            VALUES ('{{ ds }}', {{ ti.xcom_pull(task_ids='compute')['revenue_cents'] }})
        """,
    )

    compute >> load_snowflake

# Kestra flow — after migration
id: daily_revenue
namespace: prod.warehouse
description: "Migrated from Airflow DAG daily_revenue"

inputs:
  - id: run_date
    type: DATE
    defaults: "{{ execution.startDate | date('yyyy-MM-dd') }}"

tasks:
  - id: compute
    type: io.kestra.plugin.scripts.python.Script
    script: |
      from kestra import Kestra

      # ... same business logic as the PythonOperator ...
      revenue_cents = 1234500  # placeholder

      Kestra.outputs({"revenue_cents": revenue_cents})

  - id: load_snowflake
    type: io.kestra.plugin.jdbc.snowflake.Query
    url: "jdbc:snowflake://acme.snowflakecomputing.com/?warehouse=LOAD_WH"
    username: "{{ secret('SNOWFLAKE_USER') }}"
    password: "{{ secret('SNOWFLAKE_PASSWORD') }}"
    sql: |
      INSERT INTO analytics.daily_revenue(dt, revenue_cents)
      VALUES ('{{ inputs.run_date }}', {{ outputs.compute.vars.revenue_cents }})

triggers:
  - id: nightly
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 6 * * *"

# Parity harness — runs both, diffs the results
import psycopg2
from datetime import date, timedelta

def parity_check(dt: date) -> dict:
    """Compare Airflow-populated and Kestra-populated row for the same date."""
    conn = psycopg2.connect("host=analytics-db user=parity password=…")
    with conn.cursor() as cur:
        cur.execute("""
            SELECT source, revenue_cents
            FROM   analytics.daily_revenue_parity
            WHERE  dt = %s
        """, (dt,))
        rows = dict(cur.fetchall())

    airflow_val = rows.get("airflow")
    kestra_val  = rows.get("kestra")
    return {
        "date": dt,
        "airflow": airflow_val,
        "kestra":  kestra_val,
        "match":   airflow_val == kestra_val,
    }

# Run for a week; alert on any mismatch
today = date.today()
for i in range(7):
    dt = today - timedelta(days=i)
    result = parity_check(dt)
    if not result["match"]:
        alert_slack(f"Parity mismatch for {dt}: airflow={result['airflow']} kestra={result['kestra']}")

Step-by-step explanation.

1. The Airflow → Kestra mapping is largely mechanical for standard operator families: PythonOperator → Kestra Python Script, SnowflakeOperator → Kestra Snowflake Query, S3ToRedshiftOperator → Kestra S3 Upload + Redshift Query composition. The one-to-one operator translation covers ~90% of DAGs.
2. XCom-based fan-in (ti.xcom_pull(task_ids='compute')['revenue_cents']) becomes Kestra's Pebble binding ({{ outputs.compute.vars.revenue_cents }}). The semantic is identical — "read a named field produced by an upstream task"; only the syntax changes.
3. The {{ ds }} Airflow template becomes {{ inputs.run_date }} in Kestra (plus a Schedule trigger that feeds the run date). Both frameworks bind the run date; Kestra's is typed (DATE) while Airflow's is a string.
4. Custom PythonOperators wrap business logic in a Python callable; Kestra's Python Script plugin wraps the same callable in an inline script: block. Migration is literal copy-paste of the callable body plus adjusting inputs/outputs to Kestra's IPC (Kestra.outputs(...)).
5. The parity harness runs both orchestrators in parallel for 4 weeks writing to distinguishable tables (analytics.daily_revenue_airflow, analytics.daily_revenue_kestra), diffs the results daily, and alerts on any mismatch. This is the safety net that lets you turn off Airflow with confidence.

Output.

Phase	Duration	Milestone
Discovery + inventory	2 weeks	200 DAGs catalogued; migration matrix built
Parity harness build	1 week	dual-write plumbing shipped
Phase 1 (safe DAGs)	4 weeks	50 low-risk DAGs migrated; parity checked
Phase 2 (bulk migration)	8 weeks	150 DAGs migrated in waves of 20
Airflow read-only	4 weeks	Airflow scheduler paused; Kestra owns production
Airflow decommission	1 week	delete Airflow deployment; celebrate

Rule of thumb. For any Airflow → Kestra migration, run both orchestrators in parallel with a parity harness for a full month before turning off Airflow. Never cut over cold; the migration cost is manageable, but the "did we lose data" panic if something drifts is not.

Worked example — the 5-minute senior orchestrator interview answer

Detailed explanation. The senior orchestrator-comparison interview question is predictable: "Kestra vs Airflow — when do you pick each?" The 5-minute answer walks all four orchestrators, names their sweet spots, and picks based on team + integration constraints. Rehearse this once; deploy it every interview.

• Minute 1. Name all four orchestrators with their differentiator in one line each.
• Minute 2. Name the four axes (authoring, UI, plugins, execution).
• Minute 3. Kestra wins scenarios.
• Minute 4. Airflow wins scenarios; Dagster and Prefect niches.
• Minute 5. The decision heuristic and the migration cost story.

Question. Draft the 5-minute senior answer that covers all four orchestrators with their sweet spots.

Input.

Minute	Topic	Key points
1	Name the four	Kestra, Airflow, Dagster, Prefect
2	The axes	authoring, UI, plugins, execution
3	Kestra wins	polyglot + non-engineer authors + plugin fit
4	Airflow / Dagster / Prefect wins	Python-heavy / asset lineage / dynamic flows
5	Heuristic + migration cost	team-fit + ~1 eng-month per 100 DAGs

Code.

Senior orchestrator answer template (5 minutes)
==============================================

Minute 1 — name all four
  "The four modern data-orchestrators worth naming: Kestra
   (YAML-first, polyglot), Airflow (Python-first, largest community),
   Dagster (Python + software-defined assets), Prefect (Python with
   dynamic runtime graphs). All four are open-source; all four scale
   to production workloads."

Minute 2 — the four axes
  "I compare them on four axes: authoring surface (YAML vs Python),
   UI depth (authoring vs read-mostly), plugin ecosystem (JAR isolation
   vs pip conflicts), and execution model (Kafka-backed vs Celery /
   Daemon / agent-based). The pattern is Kestra + Dagster invest most
   in UI; Kestra + Airflow have the largest plugin catalogs; Airflow +
   Dagster + Prefect are Python-first authoring; Kestra is YAML-first."

Minute 3 — when Kestra wins
  "Kestra wins when the team is polyglot — Python plus SQL plus JS plus
   containers — and when non-engineers need to author schedules. The
   YAML + UI editor unblocks analytics engineers and BI folks. Plugin
   coverage is 600+ JARs across Snowflake, BigQuery, dbt, Airbyte,
   S3, GCS, Kafka. If your integration list is covered, you get JVM
   dependency isolation for free — no more 'pip install
   apache-airflow-providers-X broke Y' incidents."

Minute 4 — when Airflow / Dagster / Prefect win
  "Airflow wins when the team is Python-only and custom Python operators
   are the daily workload — deep community, mature ecosystem, most
   niche integrations. Dagster wins when software-defined asset lineage
   is a first-class UX requirement — you want the UI to be an asset
   graph. Prefect wins when flow topology is dynamic — built at
   runtime based on data or config."

Minute 5 — the heuristic + migration cost
  "The heuristic: pick per team culture, not per hype cycle. Polyglot
   team with non-engineer authors → Kestra. Python-first team with
   deep custom operators → Airflow. Asset-lineage-first team →
   Dagster. Dynamic-flow team → Prefect. Migration cost between them
   is roughly one engineer-month per 100 DAGs — mostly translating
   custom operators. Run both orchestrators in parallel with a parity
   harness for a month before decommissioning."

Step-by-step explanation.

1. Minute 1 is the framing win. Naming all four orchestrators immediately signals you've evaluated the landscape, not just used the tool the last team happened to pick. Weak candidates name one ("we use Airflow") and defend it; senior candidates name all four with differentiators.
2. Minute 2 establishes the axes framework. The four-axis comparison (authoring, UI, plugins, execution) is reusable across any orchestrator question and demonstrates you have a mental model, not just tool preferences.
3. Minute 3 makes the case for Kestra. The three keys are polyglot team, non-engineer authoring, and plugin catalog fit. Naming these three unblocks the interviewer to challenge on the third ("do you know if Snowflake is a first-class plugin?" — yes, io.kestra.plugin.jdbc.snowflake).
4. Minute 4 makes the case against Kestra where appropriate. This is the maturity signal — senior candidates admit their preferred tool loses in some scenarios. Naming the Airflow / Dagster / Prefect sweet spots shows you've thought about when not to pick Kestra.
5. Minute 5 closes with the heuristic and the migration cost. Quantifying the migration cost (~1 engineer-month per 100 DAGs) signals you've either done a migration or thought seriously about doing one. Recommending the parity-harness approach signals operational maturity.

Output.

Signal	Weak answer	Senior answer
Names all four orchestrators	rare	mandatory
Uses the four-axis framework	rare	required
Names when Kestra loses	rare	senior signal
Quantifies migration cost	rare	senior signal
Recommends parity harness	rare	senior signal

Rule of thumb. The senior orchestrator answer covers all four options with sweet spots and losses without picking a single "best" — the maturity signal is naming the trade-off, not defending a preference. Rehearse the 5-minute monologue once; deploy it every interview.

Worked example — the interview probe rubric

Detailed explanation. Beyond the "which orchestrator" question, senior Kestra interviews drill into the specifics: YAML flow shape, plugin dispatch, executor / worker split, trigger types, Pebble templating. Codifying the rubric helps you prepare answers to the follow-up probes.

• Rubric structure. 5 probes; each has a weak answer, a senior answer, and the specific detail interviewers listen for.
• Common trap. Naming Airflow features that don't exist in Kestra (Airflow variables, XCom, connections) without their Kestra equivalents.

Question. Build the probe rubric for a senior Kestra interview.

Input.

Probe	Weak answer	Senior answer
YAML flow shape	"there's a YAML file"	"id, namespace, tasks; each task has type: pointing at a plugin class"
Plugin dispatch	"there are plugins"	"JAR files loaded at boot; @plugin annotation; JVM classloader isolation"
Executor / worker split	"there are workers"	"executor schedules, worker executes; Kafka topics between them; worker groups for specialisation"
Trigger types	"cron"	"Schedule, Webhook, Flow (event-driven), Realtime (Kafka consumer)"
Pebble templating	"there are templates"	"Pebble; {{ execution.startDate }}, {{ inputs.date }}, {{ outputs.step1.vars.foo }}, {{ secret('KEY') }}"

Code.

Kestra interview probe rubric
=============================

Probe 1 — YAML flow shape
  ✗ Weak:  "you write a YAML file"
  ✓ Senior: "top-level keys id, namespace, tasks, plus optional
             triggers, inputs, outputs, labels, errors. Each task has
             a type: field pointing at a Java plugin class"

Probe 2 — Plugin dispatch
  ✗ Weak:  "there are plugins for various tools"
  ✓ Senior: "plugins are JVM JARs loaded at boot from /app/plugins/;
             each registers @Plugin-annotated task classes; JVM
             classloader isolation means no pip-style dependency
             conflicts"

Probe 3 — Executor / worker split
  ✗ Weak:  "there's a scheduler and workers"
  ✓ Senior: "5 components: executor (schedules), worker (executes),
             webserver (UI/API), scheduler (evaluates triggers),
             indexer (backfills ES). Connected by Kafka topics; state
             in Postgres. Worker groups (worker.group=<tag>) route
             specialised tasks (GPU, VPC) to specific worker fleets"

Probe 4 — Trigger types
  ✗ Weak:  "cron schedules"
  ✓ Senior: "Schedule (cron), Webhook (HTTP), Flow (event-driven from
             other flows), Realtime (streaming consumer like Kafka).
             Realtime turns Kestra into a lightweight stream processor"

Probe 5 — Pebble templating
  ✗ Weak:  "there are template variables"
  ✓ Senior: "Pebble (Jinja-like) with 4 common bindings:
             {{ execution.startDate }} for the run timestamp,
             {{ inputs.name }} for typed inputs,
             {{ outputs.taskId.vars.field }} for upstream outputs,
             {{ secret('KEY') }} for backend-pluggable secrets. Filters
             include date(), truncate(), json_encode, map()"

Step-by-step explanation.

1. Probe 1 (YAML flow shape) tests whether you've actually authored a Kestra flow. Naming the exact top-level keys (id, namespace, tasks) and the optional ones (triggers, inputs, outputs, labels, errors) signals hands-on experience.
2. Probe 2 (Plugin dispatch) tests whether you understand the JAR-vs-pip distinction. Naming @Plugin annotation and JVM classloader isolation is the specific detail that separates docs-readers from operators.
3. Probe 3 (Executor / worker) tests architectural understanding. Naming all 5 components — executor, worker, webserver, scheduler, indexer — and the Kafka + Postgres backend is the senior signal.
4. Probe 4 (Trigger types) tests coverage of the trigger palette. Weak candidates only name cron; senior candidates name Schedule + Webhook + Flow + Realtime.
5. Probe 5 (Pebble templating) tests fluency with the binding language. Naming the four most common bindings plus a few filters (date(), truncate(), json_encode, map()) signals you've read the Pebble docs.

Output.

Probe	Interviewer listens for
YAML flow shape	id / namespace / tasks / triggers
Plugin dispatch	@plugin + JVM classloader isolation
Executor / worker	5 components + Kafka + Postgres
Trigger types	Schedule + Webhook + Flow + Realtime
Pebble templating	4 bindings + 3+ filters

Rule of thumb. For any Kestra interview, memorise the probe rubric — the interviewer will drill into 3-4 of the 5 probes. Being able to name the senior answer to each without hesitation is what separates a "yes, they've used it" from "let's hire them."

Senior interview question on Kestra vs Airflow

A senior interviewer might ask: "A team wants to migrate a 500-DAG Airflow platform to Kestra. Walk me through the evaluation, the migration plan, the parity harness, and the cutover decision — including the go / no-go criteria at each phase. Also cover the residual reasons you might tell them 'stay on Airflow.'"

Solution Using a phased evaluation + parity harness + go/no-go gates + honest stay-on-Airflow criteria

Kestra migration evaluation plan (6-week evaluation, 12-week migration)
=======================================================================

Phase 0 — Team readiness (week 0)
  Go criteria:
  ✓ Team can commit 2 engineers @ 50% for 18 weeks
  ✓ Team has JVM ops experience (or willing to learn)
  ✓ Team accepts YAML-first authoring
  No-go signals:
  ✗ Team refuses YAML on aesthetic grounds
  ✗ Team has 100+ custom Python operators with no YAML equivalent
  ✗ Team has hard external SLAs during the migration window

Phase 1 — Evaluation (weeks 1-6)
  Deploy standalone Kestra + docker-compose Postgres in a sandbox VPC.
  Port 10 representative DAGs across 4 patterns:
    - simple ETL (Python + SQL)
    - fan-out (dynamic task mapping)
    - cross-DAG dependency
    - sensor-based (poll-until)
  Score each on: LOC, authoring time, UI usability, plugin coverage.
  Go criteria after Phase 1:
  ✓ All 10 DAGs port successfully within 2× the original LOC
  ✓ Plugin catalog covers ≥ 90% of existing operator usage
  ✓ 2 engineers report positive dev-loop experience
  No-go: any of the above missed → recommend stay-on-Airflow.

Phase 2 — Production deployment (weeks 7-10)
  Deploy distributed Helm cluster in prod VPC (executor + worker + …).
  Wire external RDS Postgres + MSK Kafka + managed ES.
  Stand up parity harness — dual-write to distinguishable output tables.
  Migrate 20 pilot DAGs (non-critical, well-understood).
  Go criteria after Phase 2:
  ✓ Cluster passes 4-hour load test at 2× peak
  ✓ Parity harness reports 100% match for 2 weeks on pilot DAGs
  ✓ On-call is trained on Kestra runbooks
  No-go: rollback pilot DAGs to Airflow; investigate.

Phase 3 — Bulk migration (weeks 11-16)
  Migrate remaining 480 DAGs in waves of 40 per week.
  Each wave: 3-day parity window; go/no-go on parity match.
  On-call incidents attributed to Kestra > 3/week → pause + investigate.
  Go criteria after Phase 3:
  ✓ All 500 DAGs migrated
  ✓ Parity match > 99.9% across last week
  ✓ Airflow scheduler set read-only

Phase 4 — Decommission (weeks 17-18)
  Delete Airflow deployment.
  Archive Airflow git history for reference.
  Publish post-mortem to the platform blog.

Stay-on-Airflow criteria (do NOT migrate)
  ✗ You have > 100 custom PythonOperators with no YAML equivalent
  ✗ Your team is Python-only and rejects YAML authoring
  ✗ You depend on Airflow-only integrations (some niche providers)
  ✗ Your on-call has < 6 months of runway for a big migration
  ✗ Airflow is meeting SLAs and no team pain point drives the switch

# Parity harness — dual-write to distinguishable tables
id: parity_harness_daily_revenue
namespace: prod.parity

tasks:
  - id: compute
    type: io.kestra.plugin.scripts.python.Script
    script: |
      from kestra import Kestra
      revenue_cents = 1234500  # same logic as Airflow
      Kestra.outputs({"revenue_cents": revenue_cents})

  - id: load_snowflake
    type: io.kestra.plugin.jdbc.snowflake.Query
    url: "jdbc:snowflake://…"
    username: "{{ secret('SNOWFLAKE_USER') }}"
    password: "{{ secret('SNOWFLAKE_PASSWORD') }}"
    sql: |
      INSERT INTO analytics.daily_revenue_kestra(dt, revenue_cents, source)
      VALUES ('{{ execution.startDate | date('yyyy-MM-dd') }}',
              {{ outputs.compute.vars.revenue_cents }},
              'kestra')

  - id: parity_check
    type: io.kestra.plugin.jdbc.snowflake.Query
    url: "jdbc:snowflake://…"
    username: "{{ secret('SNOWFLAKE_USER') }}"
    password: "{{ secret('SNOWFLAKE_PASSWORD') }}"
    sql: |
      SELECT
        a.dt,
        a.revenue_cents AS airflow_val,
        k.revenue_cents AS kestra_val,
        (a.revenue_cents = k.revenue_cents) AS match
      FROM   analytics.daily_revenue_airflow a
      JOIN   analytics.daily_revenue_kestra  k ON a.dt = k.dt
      WHERE  a.dt = '{{ execution.startDate | date('yyyy-MM-dd') }}'
    fetchType: FETCH

  - id: alert_if_mismatch
    type: io.kestra.plugin.core.flow.Switch
    value: "{{ outputs.parity_check.rows[0].match | string }}"
    cases:
      "true":
        - id: log_ok
          type: io.kestra.plugin.core.log.Log
          message: "Parity OK for {{ execution.startDate | date('yyyy-MM-dd') }}"
      "false":
        - id: alert
          type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
          url: "{{ secret('SLACK_URL') }}"
          payload: |
            {"text": "❌ Parity MISMATCH for {{ execution.startDate | date('yyyy-MM-dd') }}: airflow={{ outputs.parity_check.rows[0].airflow_val }} kestra={{ outputs.parity_check.rows[0].kestra_val }}"}

triggers:
  - id: daily
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 8 * * *"

Step-by-step trace.

Phase	Duration	Gate	On-fail
Team readiness	week 0	3 conditions	do not start
Evaluation	weeks 1-6	10 DAGs port, plugin coverage ≥ 90%	recommend stay-on-Airflow
Prod deployment	weeks 7-10	load test + parity + on-call training	rollback pilots
Bulk migration	weeks 11-16	parity > 99.9% per wave	pause + investigate
Decommission	weeks 17-18	Airflow read-only 2 weeks	delete

After 18 weeks, Kestra owns production and Airflow is decommissioned. Parity harness caught 3 real bugs in the pilot phase (all traced to differences in {{ ds }} vs {{ execution.startDate }} semantics — solved by defensive date formatting). On-call incident rate stabilised at parity with Airflow within 6 weeks post-decommission.

Output:

Concern	Answer
Total duration	18 weeks (6 eval + 12 migration)
Engineer investment	~2 × 50% × 18 weeks = 18 engineer-weeks
Migration cost per DAG	~1 engineer-hour
Parity harness runtime	3-day rolling window per wave
Rollback SLA per DAG	< 5 min (git revert + Kestra flow update)
Post-migration Airflow cost	zero (fully decommissioned)
Stay-on-Airflow trigger	any Phase 1 no-go criterion met

Why this works — concept by concept:

• Phased evaluation with go/no-go gates — every phase has explicit exit criteria. Phase 1 (evaluation) fails fast if plugin coverage or LOC doesn't work out; phase 2 (prod deployment) fails fast on load-test regressions; phase 3 (bulk migration) fails fast on parity misses. No blind commitment.
• Parity harness dual-write — the safety net. Both Airflow and Kestra write to distinguishable output tables; a nightly job diffs them and alerts on mismatch. This catches every semantic drift (date formatting, timezone handling, XCom → outputs mapping) before it becomes a production incident.
• Wave-based bulk migration — 40 DAGs per week with a 3-day parity window per wave. This caps the blast radius; a bad wave can be rolled back without touching earlier successful waves.
• Explicit "stay on Airflow" criteria — the mature migration plan says "here's when NOT to migrate." Senior architects earn trust by refusing to migrate when the criteria don't line up; junior architects push the migration regardless.
• Cost — 18 engineer-weeks (mostly translation + parity investigation), 12 weeks calendar time, one Kestra cluster (~30 vCPU) running in parallel with Airflow. Compared to the ongoing cost of running an aging Airflow install with Python operator sprawl, this is a one-time investment that pays back within 6-12 months. Net O(DAGs) migration cost with a bounded blast radius per wave; the eliminated cost is the "Airflow provider upgrade broke half the DAGs" incident stream.

Design
Topic — design
Design problems on orchestrator migrations

Practice →

SQL
Topic — sql
SQL problems on parity checks and reconciliation

Practice →

---

Cheat sheet — Kestra recipes

• Which orchestrator when. Kestra is the 2026 default when your team is polyglot (Python + SQL + JS + containers) and non-engineers need to author schedules — YAML + UI editor is the lowest-friction authoring surface in the space. Airflow is the answer when the team is Python-only with deep custom operators and mature niche-provider needs. Dagster is the answer when software-defined asset lineage is a first-class UX requirement. Prefect is the answer when flow topology is dynamic at runtime. Argo Workflows is the answer when K8s-native containers are the primary task type. Print the four-axis matrix (authoring / UI / plugins / execution) on a sticky note; use it in every interview.
• YAML flow skeleton. Every flow has three mandatory top-level keys: id: <unique-within-namespace>, namespace: <dot.separated.path>, tasks: [...]. Optional top-level keys: inputs: (typed parameters — STRING, DATE, INT, JSON, FILE), outputs: (re-exported task outputs), triggers: (Schedule / Webhook / Flow / Realtime), errors: (nested tasks running on failure), labels: (searchable metadata), retry: (per-flow retry policy). Every task has a type: field pointing at a Java plugin class (io.kestra.plugin.<group>.<subgroup>.<TaskType>) plus the task's type-specific properties.
• Pebble templating cheatsheet. {{ execution.startDate }} — ISO timestamp of the current run. {{ execution.id }} — unique run ID. {{ inputs.<name> }} — typed input value. {{ outputs.<taskId>.<field> }} — upstream task output; for Python Script use {{ outputs.<taskId>.vars.<name> }} for Kestra.outputs({...}) values. {{ secret('KEY') }} — pluggable secret backend. {{ vars.<key> }} — namespace variables. {{ flow.namespace }} / {{ flow.id }} — flow metadata. {{ taskrun.value }} — current loop value inside EachParallel / EachSequential. {{ trigger.value }} / {{ trigger.body }} — trigger payload. Filters: date('yyyy-MM-dd'), truncate(N), json_encode, map(attribute='x'), list, first, last, length.
• Trigger declaration snippets. Schedule (cron): type: io.kestra.plugin.core.trigger.Schedule + cron: "0 2 * * *" + timezone: UTC. Webhook: type: io.kestra.plugin.core.trigger.Webhook + key: "<random-string>" — flow runs on POST to /api/v1/executions/webhook/<flow>/<key>. Flow (event-driven): type: io.kestra.plugin.core.trigger.Flow + namespace: <parent> + flowId: <parent> + states: [SUCCESS]. Realtime (streaming): type: io.kestra.plugin.kafka.RealtimeTrigger + Kafka properties: + topic: — one message spawns one execution.
• The 5 built-in task types you use every day. Python Script: type: io.kestra.plugin.scripts.python.Script with beforeCommands: for pip install and script: for inline code; use Kestra.outputs({...}) to emit named outputs. Postgres Query: type: io.kestra.plugin.jdbc.postgresql.Query with url:, username:, password:, sql:, fetchType: FETCH | STORE. HTTP Request: type: io.kestra.plugin.core.http.Request with uri:, method:, headers:, body: — captures response as output. EachParallel: type: io.kestra.plugin.core.flow.EachParallel with value: (Pebble expression → list) + concurrent: N cap + nested tasks: per iteration. Switch: type: io.kestra.plugin.core.flow.Switch with value: (Pebble expression) + cases: map + defaults: fallback.
• Plugin install pathways. Bundled: kestra/kestra:latest ships 100+ plugins already loaded — check /plugins in the UI. Custom Docker image (production): FROM kestra/kestra:0.20.0 + COPY plugins/*.jar /app/plugins/ — bake the plugin manifest into the image, rebuild on every plugin change. Runtime install (dev only): kestra plugins install io.kestra.plugin:plugin-name:LATEST — downloads the JAR at runtime; lost on container restart. Always use the custom Docker image pathway for production so plugin state is git-tracked.
• Custom Java plugin template. @Plugin(examples = {...}) + @Schema(title="...") on a class extending Task and implementing RunnableTask<Output>. Properties annotated @PluginProperty(dynamic = true) get Pebble templating applied automatically. run(RunContext ctx) returns an Output builder. Gradle build uses kestra-processor annotation processor to generate the plugin manifest; shadow builds a fat JAR. Drop into /app/plugins/ via custom Docker image. Invoke from YAML as type: <fully-qualified-class-name> — identical shape to built-ins.
• Executor / worker docker-compose sketch. Single-node dev: postgres + kestra (mode server standalone) + shared volumes; UI on 8080. Distributed dev: postgres + kafka + elasticsearch + one container each for kestra server executor, kestra server worker, kestra server webserver, kestra server scheduler, kestra server indexer — same KESTRA_CONFIGURATION env var in every container. Production: Helm chart with per-component deployments, external RDS + MSK + managed ES, HPA on worker deployments.
• Worker groups + scaling knobs. Every worker registers with configuration.kestra.worker.group: <tag> — tasks tagged workerGroup: <tag> route to that fleet only. Use for GPU workers (ML tasks), on-prem VPC workers (locked-down DB access), container-launcher workers (Docker socket mount), standard workers (everything else). Kafka partition count on worker-tasks-<group> topic should be >= peak worker replica count. HPA on standard worker deployment scales by CPU; specialised worker fleets stay fixed. Bump kestra.executor.thread-count when executor thread pool sustained > 80%.
• The 5 Prometheus alerts every prod cluster needs. (1) KestraWorkerLagHigh — kafka_consumergroup_lag{consumergroup=~"kestra-worker-.*"} > 1000 for 5m → scale worker deployment. (2) KestraPgPoolExhausted — kestra_datasource_hikari_pool_usage > 0.8 for 10m → bump pool + investigate hoarding query. (3) KestraExecutionsFailing — failed / total > 5% for 30m → investigate top-offending namespaces. (4) KestraExecutorStarved — executor thread pool > 90% for 5m → bump threads + add replica. (5) KestraSchedulerDown — no scheduler heartbeat for 2m → check scheduler pods. All metrics are exposed on /prometheus endpoint; scrape from every component.
• Migration cost between orchestrators. Airflow → Kestra: ~1 engineer-month per 100 DAGs — mostly PythonOperator → Python Script translation, Airflow XCom → Kestra outputs.vars mapping, {{ ds }} → {{ execution.startDate }} renames. Run both in parallel with a parity harness for 4 weeks before decommissioning. Kestra → Airflow: comparable cost in reverse; usually only done if a team pivots to Python-first authoring. Kestra → Dagster: harder — requires re-architecting flows as software-defined assets, not a straight translation. Prefect → Kestra: mechanical for standard tasks; dynamic-flow Prefect patterns don't map cleanly.
• Bootstrap / secret store policy. Kestra secret store is pluggable — JDBC (Postgres row per key), HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Elasticsearch. Configure via kestra.secret.<backend> in cluster config. From YAML: {{ secret('KEY_NAME') }} — same syntax regardless of backend. Never inline secrets in YAML; every flow's git diff should be safe to share publicly. Rotate secrets by updating the backend; flows using {{ secret('KEY') }} pick up new values on next execution automatically.
• Failure semantics reminder. Executor crash → run 2+ executor replicas; Kafka rebalances; in-flight state resumes from last committed offset. Worker crash → the task fails; Kestra's per-task retry: policy handles re-execution; the worker fleet auto-scales the crashed replica. Postgres outage → executor and worker halt (they need state); healthcheck kills pods; K8s restarts on Postgres recovery. Kafka outage → executor and worker halt; recover on Kafka return; in-flight messages replay from committed offsets. Every failure mode has a bounded recovery story; the runbook is per-component, not per-flow.
• What senior interviewers score highest. Naming YAML as the authoring surface (not "some config file") in sentence one; distinguishing "polyglot on the worker" from "polyglot on the author" when comparing Kestra and Airflow; naming JVM classloader isolation as the plugin-model differentiator (not "there are plugins"); naming the 5 cluster components (executor, worker, webserver, scheduler, indexer) in order; naming all 4 trigger types (Schedule, Webhook, Flow, Realtime) — especially Flow (event-driven) and Realtime (streaming); naming the 4 Pebble bindings (execution.*, inputs.*, outputs.*.vars.*, secret()) with example filters; naming the migration cost (~1 engineer-month per 100 DAGs) and recommending the parity harness. These are the senior signals that separate architects who have run Kestra in production from candidates who have only read the marketing page.

Frequently asked questions

What is Kestra in one sentence?

Kestra is the open-source, YAML-first orchestrator built for polyglot data platforms — every workflow is a declarative YAML file with id, namespace, and tasks at the top level, each task typed by a Java plugin class (io.kestra.plugin.gcp.bigquery.Query, io.kestra.plugin.dbt.cli.DbtCLI, io.kestra.plugin.scripts.python.Script), bindings expressed in Pebble templating ({{ execution.startDate }}, {{ inputs.date }}, {{ outputs.step1.vars.foo }}), and executed by a distributed cluster (executor, worker, webserver, scheduler, indexer) on a Kafka message bus with Postgres state and optional Elasticsearch for search history. The kestra project sits between Airflow (Python-only DAGs) and Argo Workflows (K8s-native container DSL), giving polyglot teams a YAML + UI + JAR-plugin sweet spot that neither of the neighbours can match. Every senior data-engineering interview in 2026 probes Kestra because the orchestrator choice is one of the highest-lock-in decisions in a modern data platform.

Kestra vs Airflow — when do I pick each?

Default to Kestra when the team is polyglot (Python + SQL + JS + containers), non-engineers need to author schedules (analytics engineers, BI, ops), and the plugin catalog covers your integrations (Snowflake, BigQuery, dbt, Airbyte, S3, GCS, Kafka, Slack — all first-class). Kestra's YAML + UI editor is the lowest-friction authoring surface in the space, and JVM classloader isolation gives you dependency safety pip-based providers cannot match. Default to Airflow when the team is Python-only and every scheduler user is comfortable with the DAG API; when you have a large library of custom Python operators that would need Java rewrites; when you depend on Airflow-only provider integrations (some niche warehouses or ML tools); when your sensor-based waiting patterns are mature and you don't want to translate them. The kestra vs airflow decision is fundamentally about team culture (YAML vs Python authoring) and existing investment (custom operator library) — not about which tool is "better." Senior interviewers score highest on the answer that names both sweet spots and picks based on team + integration constraints rather than personal preference.

What is the Kestra plugin ecosystem?

The Kestra kestra plugin catalog is a set of 600+ drop-in JVM JAR files that register typed task classes for every mainstream data-engineering integration. Categories: warehouses (Snowflake, BigQuery, Redshift, Clickhouse, Databricks SQL, Postgres, MySQL, SQL Server, Oracle, DuckDB, Trino), lakes / storage (S3, GCS, Azure Blob, MinIO, HDFS), transforms (dbt, Airbyte, Fivetran, Meltano, Great Expectations, Soda), streams (Kafka, Pulsar, Google Pub/Sub, AWS SQS, RabbitMQ), and utilities (Slack, PagerDuty, Discord, Email, Vault, AWS Secrets Manager, GCP Secret Manager). Plugins are drop-in JARs: bake them into a custom Docker image (FROM kestra/kestra:0.20.0 + COPY plugin.jar /app/plugins/), and the JVM loads them at boot. From YAML, invocation is type: io.kestra.plugin.<group>.<subgroup>.<TaskType> — identical shape whether the plugin ships in the base image, is a third-party plugin, or is a custom-built plugin you wrote in Java. Compared to Airflow's Python-package provider model, the JAR model gives JVM-classloader-level dependency isolation — you never fight pip for a version pin across two providers.

Can Kestra run Python flows?

Yes — the Python plugin (io.kestra.plugin.scripts.python.Script) runs Python code as a first-class task type, and it's arguably the most-used single task type across the Kestra catalog. Inline Python: script: | block with the code as YAML block scalar; beforeCommands: [pip install requests] for dependencies; Kestra.outputs({"key": value}) (from the kestra Python client) emits named outputs to downstream tasks via the {{ outputs.<taskId>.vars.<key> }} Pebble binding. File-based Python: mount your Python project via namespaceFiles: or use inputFiles: to inject code. Container-based Python: io.kestra.plugin.docker.Run or io.kestra.plugin.kubernetes.PodCreate for full container isolation with python:3.12-slim as the image. The polyglot promise is that the same flow file can compose Python tasks alongside SQL Query tasks (io.kestra.plugin.jdbc.postgresql.Query), HTTP Request tasks, dbt CLI tasks, and Kafka Produce tasks — each task type is a first-class plugin, none is a second-class citizen.

How does the executor / worker split work?

Kestra's cluster is composed of 5 stateless components sharing durable backends (Postgres + Kafka + optional Elasticsearch): executor schedules tasks — reads flow definitions from Postgres, publishes task-run requests to a Kafka topic, tracks execution state; worker consumes task-run requests from Kafka, runs them (Python, SQL, HTTP, plugin), publishes results back; webserver serves the UI + REST API; scheduler evaluates triggers (cron, webhook, flow, realtime) and publishes execution-start requests; indexer backfills execution results into Elasticsearch for the UI's history / search views. The split lets you scale each component independently — a morning batch surge scales workers via HPA without touching the executor; a scheduling backlog gets more executor replicas without adding worker capacity. Worker groups (worker.group: <tag>) route specialised tasks to specialised worker fleets — GPU workers for ML training, on-prem-VPC workers for locked-down DB access, standard workers for everything else. Postgres holds durable state; Kafka carries the messages; Elasticsearch (optional) indexes history. Executor / scheduler / webserver are stateless (state lives in Postgres + Kafka), so any of them can crash without data loss — Kafka rebalances and in-flight state resumes from the last committed offset.

Is Kestra production-ready in 2026?

Yes — Kestra reached general availability in the early 2020s and by 2026 is running in production at hundreds of teams from startups to Fortune 500s, including well-known data platform teams (LiveRamp, Leroy Merlin, Sopra Steria, and many others). The open-source edition (Apache 2.0) covers all core functionality — flows, plugins, executor / worker cluster, RBAC, secret store, UI editor. The Enterprise Edition adds SAML/OIDC SSO, advanced RBAC (namespace-scoped grants), audit logs, high-availability options, and priority support — the standard "OSS-plus-EE" split. The remaining rough edges in 2026 are largely around ecosystem maturity: some niche integrations (specific legacy warehouses, older ML tools) have Airflow providers but no Kestra plugin yet; the Java-based custom-plugin path is less familiar than Python-based Airflow custom operators for teams without JVM experience; the community, while growing rapidly, is smaller than Airflow's. None of these are blockers — they are the "adopt a newer tool" trade-offs any senior architect weighs. The interviewer's question here is almost always "is it mature enough to bet on" — the 2026 answer is unambiguously yes, with the caveat that you must run it operationally the way you would any other tier-1 data platform component.

Practice on PipeCode

• Drill the SQL practice library → for the query, join, aggregation, and window-function problems senior interviewers use to test Kestra-orchestrated SQL patterns.
• Rehearse system design against the design practice library → for orchestrator architecture, plugin-ecosystem, executor / worker split, and Airflow-migration scenarios that mirror the Kestra senior design interview.
• Sharpen the streaming axis with the streaming practice library → for Realtime-trigger + Kafka-consumer + upsert-with-idempotency problems and the multi-source coordination patterns typical of Kestra streaming flows.
• Warm up on the aggregation practice library → for the group-by / window-function / rolling-metric shapes that dominate the Snowflake / BigQuery / Postgres queries Kestra flows most often run.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the Kestra decision matrix, the 5-component cluster architecture, and the 4-trigger palette against real graded inputs.

Lock in kestra muscle memory

Docs explain what Kestra is. PipeCode drills explain when it wins — when YAML-first authoring unblocks non-engineers, when JAR-based plugins eliminate pip conflicts, when the executor / worker split scales cleanly under morning batch surges, when Realtime triggers make Kestra a lightweight streaming processor, when a 200-DAG Airflow migration earns its parity-harness safety net. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the orchestrator design trade-offs senior data engineers actually face.

Practice SQL problems →
Practice design problems →