Argo Workflows + Argo Events for Data Pipelines: K8s-Native Orchestration

argo workflows data pipelines is the Kubernetes-native orchestrator every platform team eventually evaluates when Airflow's Python DAG model starts feeling heavyweight or when the team is already all-in on Kubernetes CRDs and prefers YAML declarative resources over Python code. Every DE eventually picks an orchestrator; knowing when Argo Workflows wins over Airflow (K8s-first shops with heavy containerised workloads, ML training pipelines, CI/CD-style workflows) and when Airflow or Dagster wins (Python-first teams, mature provider ecosystem, asset-aware lineage) is what separates a senior platform engineer's decision from a mid-level one's cargo-culting.

The tour walks the five pillars — (1) the Workflow CRD anatomy with dag: templates for parallel dependencies vs steps: templates for sequential pipelines, parameters + artifacts + retry policies + timeouts, (2) Argo Events with EventSource (webhook, kafka, s3, calendar, github, sns) + Sensor for filtering and dispatching to Workflows, (3) the compare-and-contrast with Airflow (Python DAGs, mature ecosystem) and Dagster (asset-first, modern), (4) production patterns for WorkflowTemplate (reusable templates), CronWorkflow (scheduled), retry policies, artifact repositories, and (5) GitOps deploy via Argo CD ApplicationSet + namespace-per-env RBAC + ResourceQuota for multi-tenant isolation. Every section ships a Solution-Tail interview answer — code, trace, output, why-this-works with __concept__ underlines.

Practice on SQL library →, SQL optimization drills →, and SQL join drills →.

---

On this page

• Why K8s-native orchestration matters
• Argo Workflows DAG anatomy
• Argo Events triggers
• Compared to Airflow / Dagster
• GitOps + patterns
• Cheat sheet — Argo recipe list
• Frequently asked questions
• Practice on PipeCode

---

1. Why K8s-native orchestration matters

The argo workflows mental model — K8s CRDs as orchestration primitives

The one-sentence invariant: Argo Workflows treats every workflow as a Kubernetes Custom Resource — a Workflow YAML manifest is applied to the cluster, the Argo controller reconciles it into driver + step pods, monitors lifecycle, retries on failure, and materialises artifacts to S3 / GCS; the "orchestrator" is the K8s controller itself, so scaling, isolation, monitoring, and RBAC are all handled by the same primitives you use for the rest of your K8s platform.

Where Argo Workflows shows up in DE.

• Kubernetes-first data platforms. Team already on K8s + YAML; Argo feels natural.
• Containerised ML training pipelines. Each stage is a container (preprocess, train, evaluate); Argo runs them as DAG.
• Event-driven pipelines. S3 upload triggers job via Argo Events.
• Multi-tenant orchestration. Namespace isolation + ResourceQuota per team.
• CI/CD-style workflows. Test → build → deploy chains — where Argo replaces Jenkins.
• Ad-hoc backfills. One-shot Workflow spawned via argo submit.

The Argo family.

• Argo Workflows — DAG execution engine.
• Argo Events — event-driven triggers.
• Argo CD — GitOps for K8s manifests.
• Argo Rollouts — progressive delivery.

What senior interviewers actually probe.

• When Argo vs Airflow. K8s-native vs Python-first.
• Workflow CRD vs Airflow DAG. YAML declarative vs Python code.
• Argo Events flow. EventSource → Sensor → Workflow trigger.
• DAG vs steps template. Parallel dependencies vs sequential.
• Parameters + artifacts. Small values vs large data between steps.
• Retry + timeout policies. activeDeadlineSeconds, retryStrategy.
• ArtifactRepository. S3 / GCS as central store.
• WorkflowTemplate. Reusable base for multiple workflows.
• CronWorkflow. Scheduled workflows.
• RBAC per namespace. Multi-tenant isolation.
• ResourceQuota. Per-namespace resource caps.

The install.

kubectl create namespace argo
kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v3.6.0/install.yaml

# Or via Helm
helm repo add argo https://argoproj.github.io/argo-helm
helm install argo-workflows argo/argo-workflows -n argo --create-namespace

Worked example — a first Workflow

Code.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: hello-world-
  namespace: argo
spec:
  entrypoint: main
  templates:
    - name: main
      container:
        image: alpine:3.19
        command: [echo]
        args: ["Hello from Argo Workflows"]

Submit + observe.

argo submit -n argo hello.yaml
argo list -n argo
argo logs -n argo @latest

Rule of thumb. Every Argo workflow is a Workflow CRD; argo submit applies it.

Worked example — a 3-step ETL DAG

Code.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: etl-
spec:
  entrypoint: main
  templates:
    - name: main
      dag:
        tasks:
          - name: extract
            template: run-container
            arguments:
              parameters: [{name: cmd, value: "extract.py"}]
          - name: transform
            template: run-container
            dependencies: [extract]
            arguments:
              parameters: [{name: cmd, value: "transform.py"}]
          - name: load
            template: run-container
            dependencies: [transform]
            arguments:
              parameters: [{name: cmd, value: "load.py"}]

    - name: run-container
      inputs:
        parameters: [{name: cmd}]
      container:
        image: myco/etl:1.2
        command: [python]
        args: ["{{inputs.parameters.cmd}}"]

Rule of thumb. DAG template with dependencies: per task creates the execution graph.

Common beginner mistakes

• Using Airflow syntax expecting Argo compatibility.
• Missing entrypoint — Argo doesn't know where to start.
• No artifact repository — steps can't share large data.
• No retry policy — flaky tasks fail permanently.
• Root user in container — K8s PSP rejects.

argo workflows interview question on translating an Airflow DAG

A senior interviewer asks: "You have an Airflow DAG with 5 tasks (fetch, validate, transform, load, alert). Translate to Argo Workflows."

Solution Using Workflow CRD with DAG template

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: daily-etl-
spec:
  entrypoint: main
  arguments:
    parameters:
      - name: date
        value: "{{workflow.creationTimestamp}}"
  templates:
    - name: main
      dag:
        tasks:
          - name: fetch
            template: task
            arguments: {parameters: [{name: cmd, value: "fetch.py --date {{workflow.parameters.date}}"}]}

          - name: validate
            template: task
            dependencies: [fetch]
            arguments: {parameters: [{name: cmd, value: "validate.py"}]}

          - name: transform
            template: task
            dependencies: [validate]
            arguments: {parameters: [{name: cmd, value: "transform.py"}]}

          - name: load
            template: task
            dependencies: [transform]
            arguments: {parameters: [{name: cmd, value: "load.py"}]}

          - name: alert
            template: task
            dependencies: [load]
            arguments: {parameters: [{name: cmd, value: "alert.py"}]}

    - name: task
      inputs:
        parameters: [{name: cmd}]
      retryStrategy:
        limit: 3
        retryPolicy: OnFailure
        backoff: {duration: 30s, factor: 2}
      container:
        image: myco/etl:1.2
        command: [sh, -c]
        args: ["python {{inputs.parameters.cmd}}"]

Output: Airflow-equivalent DAG in YAML.

Why this works — concept by concept:

• DAG template with tasks + dependencies — expresses the same graph as Airflow.
• Reusable task template — DRY; parameterised by cmd.
• retryStrategy at template level — every task gets 3 retries with backoff.
• workflow.parameters.date — workflow-scoped arg propagated to all tasks.
• Container image — same containerised task as Airflow's KubernetesPodOperator.

SQL
Topic — SQL
SQL practice library

Practice →

SQL Topic — optimization SQL optimization drills

Practice →

---

2. Argo Workflows DAG anatomy

Workflow CRD — templates, parameters, artifacts, retry, timeout

The mental model in one line: an Argo Workflow YAML consists of a spec.entrypoint naming the starting template, a list of templates each of which is either a container (runs a pod), a dag (declares tasks + dependencies), a steps (sequential list with optional parallel arrays), a script (inline shell/python), a resource (kubectl apply), or a suspend (wait for signal); every template can declare inputs.parameters, inputs.artifacts, outputs.parameters, outputs.artifacts, and a retryStrategy — the entire pipeline is one YAML file that can be templated, parameterised, and reused.

Slot 1 — dag: template.

templates:
  - name: main
    dag:
      tasks:
        - name: A
          template: run
        - name: B
          template: run
          dependencies: [A]
        - name: C
          template: run
          dependencies: [A]
        - name: D
          template: run
          dependencies: [B, C]

A → [B, C] parallel → D.

Slot 2 — steps: template.

templates:
  - name: main
    steps:
      - - name: A
          template: run
      - - name: B
          template: run
        - name: C
          template: run
      - - name: D
          template: run

Sequential list of parallel arrays. Outer - = sequential; inner - = parallel.

Slot 3 — parameters.

templates:
  - name: main
    inputs:
      parameters:
        - name: message
          default: "hello"
    container:
      image: alpine
      command: [echo]
      args: ["{{inputs.parameters.message}}"]

Slot 4 — artifacts.

templates:
  - name: producer
    outputs:
      artifacts:
        - name: result
          path: /tmp/result.json
          s3:
            endpoint: s3.amazonaws.com
            bucket: myco-artifacts
            key: "{{workflow.uid}}/result.json"
    container: ...

  - name: consumer
    inputs:
      artifacts:
        - name: result
          path: /tmp/input.json
    container: ...

Artifact repository config (default global):

apiVersion: v1
kind: ConfigMap
metadata:
  name: artifact-repositories
data:
  default-v1: |
    s3:
      bucket: myco-artifacts
      endpoint: s3.amazonaws.com
      accessKeySecret: {name: aws-creds, key: id}
      secretKeySecret: {name: aws-creds, key: key}

Slot 5 — retry strategy.

templates:
  - name: run
    retryStrategy:
      limit: 3
      retryPolicy: OnFailure   # or Always, OnError, OnTransientError
      backoff:
        duration: 30s
        factor: 2
        maxDuration: 10m

Slot 6 — timeout.

spec:
  activeDeadlineSeconds: 3600   # 1 hour whole workflow
  templates:
    - name: run
      activeDeadlineSeconds: 600   # per-template 10 min
      container: ...

Slot 7 — resource limits.

container:
  image: myco/etl:1.2
  resources:
    requests: {cpu: 500m, memory: 1Gi}
    limits: {cpu: 2, memory: 4Gi}

Slot 8 — conditional execution (when).

tasks:
  - name: production_only
    template: run
    when: "{{workflow.parameters.env}} == prod"

Slot 9 — loop / withParam.

tasks:
  - name: process
    template: run
    arguments: {parameters: [{name: item, value: "{{item}}"}]}
    withParam: '[{"id": 1}, {"id": 2}, {"id": 3}]'

Runs process for each item in parallel.

Slot 10 — script template.

templates:
  - name: py-script
    script:
      image: python:3.11-slim
      command: [python]
      source: |
        import sys
        print(f"Hello from Python, args={sys.argv}")

Inline Python; no separate image needed.

Common beginner mistakes

• Missing entrypoint — workflow doesn't know where to start.
• Wrong indentation on dependencies — YAML parse error.
• No retryStrategy — transient failures halt pipeline.
• No activeDeadlineSeconds — runaway workflows.
• Artifact repository not configured — outputs vanish.

Worked example — parameters + artifacts + retry

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: pipeline-
spec:
  entrypoint: main
  arguments:
    parameters:
      - name: date
        value: "2026-07-12"
  templates:
    - name: main
      dag:
        tasks:
          - name: extract
            template: extract-step
            arguments: {parameters: [{name: date, value: "{{workflow.parameters.date}}"}]}
          - name: transform
            template: transform-step
            dependencies: [extract]
            arguments:
              artifacts: [{name: raw, from: "{{tasks.extract.outputs.artifacts.raw}}"}]
          - name: load
            template: load-step
            dependencies: [transform]
            arguments:
              artifacts: [{name: transformed, from: "{{tasks.transform.outputs.artifacts.transformed}}"}]

    - name: extract-step
      inputs: {parameters: [{name: date}]}
      outputs:
        artifacts: [{name: raw, path: /tmp/raw.parquet}]
      retryStrategy: {limit: 3, backoff: {duration: 30s, factor: 2}}
      container:
        image: myco/extract:1.0
        args: ["--date", "{{inputs.parameters.date}}", "--out", "/tmp/raw.parquet"]

    - name: transform-step
      inputs:
        artifacts: [{name: raw, path: /tmp/raw.parquet}]
      outputs:
        artifacts: [{name: transformed, path: /tmp/out.parquet}]
      container:
        image: myco/transform:1.0

    - name: load-step
      inputs:
        artifacts: [{name: transformed, path: /tmp/data.parquet}]
      container:
        image: myco/load:1.0

Rule of thumb. Parameters for small values; artifacts for files; retry per step; declare inputs + outputs.

Worked example — parallel loop with withParam

templates:
  - name: process-all
    dag:
      tasks:
        - name: process
          template: run
          arguments:
            parameters: [{name: batch_id, value: "{{item.id}}"}]
          withParam: |
            [
              {"id": "batch-1"},
              {"id": "batch-2"},
              {"id": "batch-3"}
            ]

Rule of thumb. withParam fans out N parallel tasks with per-item parameters.

Worked example — resource template for kubectl apply

templates:
  - name: apply-configmap
    resource:
      action: apply
      manifest: |
        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: dynamic-config
        data:
          value: "{{workflow.parameters.value}}"

Uses Argo to kubectl apply arbitrary K8s manifests.

argo workflows interview question on artifact flow

A senior interviewer asks: "Design a 3-step pipeline where step 1 produces a 10 GB file, step 2 transforms it, step 3 loads it. Where's the file stored between steps?"

Solution Using S3 artifact repository

# Configure artifact repository once (ConfigMap)
apiVersion: v1
kind: ConfigMap
metadata:
  name: artifact-repositories
  namespace: argo
data:
  default-v1: |
    s3:
      bucket: myco-argo-artifacts
      endpoint: s3.amazonaws.com
      accessKeySecret: {name: aws-creds, key: id}
      secretKeySecret: {name: aws-creds, key: key}

# Workflow references artifacts by name; storage transparent
templates:
  - name: step1
    outputs:
      artifacts:
        - name: bigfile
          path: /tmp/output.parquet   # Argo uploads to S3

  - name: step2
    inputs:
      artifacts:
        - name: bigfile
          path: /tmp/input.parquet    # Argo downloads from S3

Why this works — concept by concept:

• S3 artifact repository — central store for large data between steps.
• Transparent upload/download — Argo handles the S3 sync.
• No shared volume required — each step is independent pod.
• Retention — set S3 lifecycle policy for cleanup.
• Cost — S3 storage per GB; negligible for typical pipelines.

SQL
Topic — SQL
SQL practice library

Practice →

SQL Topic — joins SQL join drills

Practice →

---

3. Argo Events triggers

EventSource → Sensor → Workflow — event-driven pipeline architecture

The mental model in one line: Argo Events is a separate CRD-based library that receives external events (webhook, Kafka message, S3 object create, cron, GitHub push, Slack) via an EventSource, applies filtering/routing via a Sensor, and triggers a Workflow (or other K8s resource) — the pattern lets you compose "event → filter → orchestrate" pipelines entirely in YAML with no polling code.

Slot 1 — install.

kubectl create ns argo-events
kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/master/manifests/install.yaml

kubectl apply -n argo-events -f - <<EOF
apiVersion: argoproj.io/v1alpha1
kind: EventBus
metadata:
  name: default
spec:
  jetstream:
    version: latest
EOF

Slot 2 — EventSource kinds.

• webhook — HTTP endpoint.
• kafka — Kafka topic consumer.
• sqs — AWS SQS.
• sns — AWS SNS.
• s3 — S3 object create/delete.
• github — GitHub webhook.
• slack — Slack events API.
• redisStream — Redis Streams.
• calendar — cron / interval.

Slot 3 — webhook EventSource.

apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: webhook-source
spec:
  service:
    ports: [{port: 12000, targetPort: 12000}]
  webhook:
    order-received:
      port: "12000"
      endpoint: /order
      method: POST

Slot 4 — Kafka EventSource.

spec:
  kafka:
    orders-topic:
      url: kafka.default.svc:9092
      topic: orders
      partition: "0"
      consumerGroup:
        name: argo-events
      jsonBody: true

Slot 5 — Sensor.

apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: order-sensor
spec:
  dependencies:
    - name: order-received
      eventSourceName: webhook-source
      eventName: order-received
      filters:
        data:
          - path: body.total
            type: number
            comparator: ">"
            value: ["1000"]

  triggers:
    - template:
        name: trigger-workflow
        k8s:
          operation: create
          source:
            resource:
              apiVersion: argoproj.io/v1alpha1
              kind: Workflow
              metadata:
                generateName: process-order-
              spec:
                entrypoint: main
                arguments:
                  parameters:
                    - name: order-id
                      value: ""
                templates:
                  - name: main
                    container:
                      image: myco/order-processor:1.0
                      args: ["--order-id", "{{workflow.parameters.order-id}}"]
          parameters:
            - src:
                dependencyName: order-received
                dataKey: body.id
              dest: spec.arguments.parameters.0.value

Slot 6 — event filtering.

• data: filter — inspect payload fields.
• ctx: filter — inspect event metadata.
• Comparators. >, <, >=, <=, ==, !=, contains, regex.
• Multiple filters — AND semantics.

Slot 7 — S3 event triggers.

spec:
  s3:
    input-bucket:
      bucket:
        name: myco-inputs
      accessKey: {name: aws-creds, key: id}
      secretKey: {name: aws-creds, key: key}
      events:
        - s3:ObjectCreated:Put

New object in bucket → trigger.

Slot 8 — calendar (cron).

spec:
  calendar:
    daily-etl:
      schedule: "0 3 * * *"

Same as CronWorkflow but via events model.

Slot 9 — passing event data to workflow.

triggers:
  - template:
      k8s:
        source: {resource: {...}}
        parameters:
          - src:
              dependencyName: kafka-event
              dataKey: body.order_id
            dest: spec.arguments.parameters.0.value

Extracts event field → workflow parameter.

Slot 10 — retry + backoff on trigger.

triggers:
  - template:
      name: my-trigger
      retryStrategy:
        steps: 3
        duration: 30s
        factor: 2
      k8s: {...}

Common beginner mistakes

• Missing EventBus — Sensors don't fire.
• Wrong filter path — event doesn't match.
• Not exposing webhook via Ingress — external can't reach.
• No RBAC for Sensor to create Workflow — permission denied.
• Kafka consumer group same for two Sensors — race condition.

Worked example — S3-triggered ETL

apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: s3-drops
spec:
  s3:
    input-bucket:
      bucket: {name: myco-drops}
      accessKey: {name: aws-creds, key: id}
      secretKey: {name: aws-creds, key: key}
      events: ["s3:ObjectCreated:Put"]
      filter: {prefix: "incoming/", suffix: ".parquet"}
---
apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: s3-etl-sensor
spec:
  dependencies:
    - name: file-arrived
      eventSourceName: s3-drops
      eventName: input-bucket
  triggers:
    - template:
        name: run-etl
        k8s:
          operation: create
          source:
            resource:
              apiVersion: argoproj.io/v1alpha1
              kind: Workflow
              spec:
                entrypoint: process
                arguments:
                  parameters: [{name: s3_key, value: ""}]
                templates: [...]
          parameters:
            - src: {dependencyName: file-arrived, dataKey: object.key}
              dest: spec.arguments.parameters.0.value

Rule of thumb. File-arrival triggers = S3 EventSource + Sensor + Workflow.

Worked example — Kafka-triggered pipeline

Similar structure with kafka: EventSource.

Rule of thumb. Kafka triggers = same 3-CRD pattern; scale via consumer groups.

Worked example — filtered event

dependencies:
  - name: high-value
    eventSourceName: kafka-source
    eventName: orders
    filters:
      data:
        - path: body.total
          type: number
          comparator: ">"
          value: ["1000"]

Only orders > $1K trigger.

Rule of thumb. Filter at Sensor to reduce workflow load.

argo events interview question on end-to-end flow

A senior interviewer asks: "Design an event-driven pipeline: a file arrives in S3, workflow processes it, notification on completion."

Solution Using EventSource + Sensor + Workflow + notification trigger

Complete flow: S3 EventSource → Sensor filter → Workflow → notification trigger to Slack.

SQL
Topic — SQL
SQL practice library

Practice →

SQL Topic — optimization SQL optimization drills

Practice →

---

4. Compared to Airflow / Dagster

K8s-native YAML vs Python DAG vs asset-first — the three orchestrator paradigms

The mental model in one line: Argo Workflows expresses pipelines as Kubernetes CRDs in YAML — K8s-native, declarative, container-first, ideal for teams already on K8s with heavy containerised workloads; Airflow expresses pipelines as Python DAGs — Python-native, mature provider ecosystem, best for Python data teams; Dagster expresses pipelines as software-defined assets — asset-first with lineage, best for modern data platforms — each wins in specific team contexts, and all three can coexist in a large org.

Slot 1 — the trade-off matrix.

Aspect	Argo Workflows	Airflow	Dagster
Author language	YAML CRD	Python	Python
K8s native	✅	Requires KubernetesExecutor	Requires K8s launcher
UI	Basic	Mature	Modern
Event triggers	Argo Events	Sensors + external	Sensors native
Asset-first	❌	❌	✅
Ecosystem	Small	Huge	Growing
Multi-tenancy	Namespaces	Airflow tenants	Deployments
Best for	K8s DevOps-heavy	Python DE teams	Modern platforms

Slot 2 — when Argo wins.

• Container-heavy workflows (ML training pipelines with N stages of different containers).
• Team already on K8s + YAML.
• Simple to moderate DAGs; complex dependencies fine.
• CI/CD-style workflows.
• Multi-tenant K8s cluster with namespace isolation.

Slot 3 — when Airflow wins.

• Python-first team.
• Many small DAG tasks with rich provider integrations (BigQuery, Snowflake, S3).
• Mature ecosystem — hooks / operators / sensors for everything.
• Asset lineage less important than task orchestration.
• Analytics engineering teams with dbt models.

Slot 4 — when Dagster wins.

• Asset-first mental model — pipelines as directed graph of data assets.
• Strong typing throughout.
• Modern developer experience.
• Data quality checks as first-class primitives.
• Modern data platform teams starting fresh.

Slot 5 — Argo YAML vs Airflow Python.

# Argo — YAML
- name: extract
  template: run-container
  dependencies: []
- name: transform
  template: run-container
  dependencies: [extract]
- name: load
  template: run-container
  dependencies: [transform]

# Airflow — Python
extract = BashOperator(task_id="extract", bash_command="...", dag=dag)
transform = BashOperator(task_id="transform", bash_command="...", dag=dag)
load = BashOperator(task_id="load", bash_command="...", dag=dag)
extract >> transform >> load

Same result; different paradigms.

Slot 6 — feature comparison.

Feature	Argo	Airflow	Dagster
Scheduled	CronWorkflow	Schedule	Schedule
Event-driven	Argo Events	Sensors	Sensors
Retry	Yes	Yes	Yes
Backfill	Manual	Native	Native
Lineage	Manual	Limited	Native
Testing	K8s integration	pytest DAG	pytest asset
Metrics	Prometheus	Prometheus	Prometheus
UI	Basic	Full	Full

Slot 7 — deployment complexity.

• Argo — 3 CRDs; requires K8s cluster.
• Airflow — 5-10 components (scheduler, webserver, workers, metadata DB, triggerer); Helm chart handles it.
• Dagster — daemon + webserver + user code deployments; more moving parts.

Slot 8 — operator complexity.

• Argo — K8s knowledge required; YAML learning curve.
• Airflow — Python knowledge; DAG author autonomy; DevOps for platform.
• Dagster — Python + Dagster concepts.

Slot 9 — cost of switching.

• Argo → Airflow: rewrite YAML DAGs to Python — significant.
• Airflow → Argo: rewrite Python DAGs to YAML — significant.
• Argo → Dagster: Argo → Dagster asset — major refactor.
• Rule — pick once, commit.

Slot 10 — hybrid patterns.

Some orgs use both — Airflow for Python DE workflows, Argo for ML training pipelines. Namespaces isolate. Slack alerts unified.

Common beginner mistakes

• Picking based on hype rather than team fit.
• Underestimating YAML complexity for large Argo pipelines.
• Overestimating Airflow K8s integration (KubernetesExecutor is not the same as native).
• Ignoring Dagster's asset lineage benefits.

Worked example — same pipeline in all three

Same 3-step ETL in Argo YAML, Airflow Python, Dagster Python.

Argo.

templates:
  - name: main
    dag:
      tasks:
        - {name: extract, template: run}
        - {name: transform, template: run, dependencies: [extract]}
        - {name: load, template: run, dependencies: [transform]}

Airflow.

extract >> transform >> load

Dagster.

@asset
def raw(): return load_from_source()

@asset
def transformed(raw): return transform_data(raw)

@asset
def loaded(transformed): return write_to_warehouse(transformed)

Rule of thumb. Same DAG, three paradigms.

Worked example — team decision framework

Decision tree:

• Python team, mature stack → Airflow.
• K8s team, containers everywhere → Argo.
• Data platform starting fresh, asset lineage matters → Dagster.
• Small team, simple pipelines → any works.

Rule of thumb. Team fit > tool features.

Worked example — running Airflow DAG that submits Argo Workflow

from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator

submit_argo = KubernetesPodOperator(
    task_id="submit_argo",
    image="argoproj/argocli:v3.6.0",
    cmds=["argo"],
    arguments=["submit", "-n", "argo", "my-workflow.yaml", "--wait"],
    is_delete_operator_pod=True,
)

Airflow orchestrates but delegates to Argo for K8s-heavy workflows.

Rule of thumb. Hybrid pattern — Airflow for the high-level DAG, Argo for the containerised heavy work.

argo vs airflow interview question on team decision

A senior interviewer asks: "Your team is 60% Python DE, 40% platform / SRE. You're on K8s. Pick an orchestrator."

Solution Using Airflow + KubernetesExecutor as compromise

Airflow with KubernetesExecutor:

• DE team writes DAGs in Python.
• Platform team runs on K8s.
• KubernetesPodOperator delegates heavy work to containers.
• No YAML DAG author.
• Familiar to both.

Why this works — concept by concept:

• Python DAGs — DE team writes normally.
• KubernetesExecutor — every task in K8s pod; platform-friendly.
• KubernetesPodOperator — heavy tasks in specific containers.
• Helm chart — platform manages.
• Familiar — Airflow is well-known.

SQL
Topic — SQL
SQL practice library

Practice →

SQL Topic — optimization SQL optimization drills

Practice →

---

5. GitOps + patterns

Argo CD deploy + WorkflowTemplate + CronWorkflow + namespace isolation

The mental model in one line: deploy Argo Workflows itself via Argo CD sync from git; store WorkflowTemplates in git so they're versioned + reviewed; CronWorkflow for schedules replaces Airflow's schedule_interval; per-tenant namespaces + RBAC + ResourceQuota + NetworkPolicy isolate teams; the entire orchestration surface is Kubernetes-native GitOps.

Slot 1 — Argo CD deploys Argo Workflows.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argo-workflows
spec:
  source:
    repoURL: [e-mail]:myco/platform.git
    path: charts/argo-workflows
    helm:
      valueFiles: [values-prod.yaml]
  destination:
    server: https://kubernetes.default.svc
    namespace: argo
  syncPolicy:
    automated: {prune: true, selfHeal: true}

Slot 2 — WorkflowTemplate for reuse.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: etl-template
spec:
  entrypoint: main
  templates:
    - name: main
      dag: {...}
    - name: task
      inputs: {parameters: [{name: cmd}]}
      container: {...}

Reference from a Workflow:

kind: Workflow
spec:
  workflowTemplateRef:
    name: etl-template

Slot 3 — CronWorkflow.

apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: daily-etl
spec:
  schedule: "0 4 * * *"
  timezone: "UTC"
  concurrencyPolicy: Forbid   # Forbid, Replace, Allow
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  workflowSpec:
    entrypoint: main
    templates: [...]

Slot 4 — namespace-per-env.

argo-dev/
argo-staging/
argo-prod/
argo-team-analytics/
argo-team-ml/

RBAC per namespace; ResourceQuota; NetworkPolicy.

Slot 5 — ResourceQuota.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: workflow-quota
  namespace: argo-team-analytics
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    pods: "50"

Prevents one team's runaway from starving cluster.

Slot 6 — RBAC.

kind: Role
metadata:
  name: workflow-editor
rules:
  - apiGroups: [argoproj.io]
    resources: [workflows, workflowtemplates, cronworkflows]
    verbs: [create, get, list, watch, delete, update]
---
kind: RoleBinding
metadata:
  name: analytics-team-workflows
subjects:
  - kind: Group
    name: analytics-team
roleRef:
  kind: Role
  name: workflow-editor

Slot 7 — Argo CD ApplicationSet for multi-team.

kind: ApplicationSet
metadata:
  name: team-workflows
spec:
  generators:
    - list:
        elements:
          - {team: analytics}
          - {team: ml}
          - {team: finance}
  template:
    metadata:
      name: workflows-{{ team }}
    spec:
      source:
        path: teams/{{ team }}/workflows

Each team's git subdirectory syncs to their namespace.

Slot 8 — artifact repository per team.

apiVersion: v1
kind: ConfigMap
metadata:
  name: artifact-repositories
  namespace: argo-team-analytics
data:
  default-v1: |
    s3:
      bucket: myco-argo-analytics
      ...

Per-team S3 buckets; separate lifecycle policies.

Slot 9 — monitoring + alerting.

• Prometheus ServiceMonitor for Argo controller metrics.
• Alertmanager rules for workflow failures.
• Slack notification via Argo Events + trigger.

Slot 10 — versioning.

• WorkflowTemplates versioned by name suffix (etl-v1, etl-v2).
• Git tags for platform releases.
• Argo CD tracks sync history for rollback.

Common beginner mistakes

• Not using WorkflowTemplate — copy-paste per workflow.
• No CronWorkflow — running manually.
• Single namespace for all teams — blast radius.
• No ResourceQuota — one team hogs cluster.
• No monitoring — silent failures.

Worked example — team GitOps structure

platform-config/
├── argocd-apps/
│   ├── argo-workflows-app.yaml
│   ├── team-workflows-appset.yaml
│   └── ...
├── charts/
│   └── argo-workflows/
│       └── values-prod.yaml
├── teams/
│   ├── analytics/
│   │   ├── workflows/
│   │   │   ├── etl-template.yaml
│   │   │   └── daily-cron.yaml
│   │   ├── namespace.yaml
│   │   └── quota.yaml
│   ├── ml/
│   │   └── ...

Rule of thumb. Team gets a subdirectory; Argo CD syncs to their namespace.

Worked example — WorkflowTemplate library

kind: WorkflowTemplate
metadata:
  name: standard-etl
spec:
  arguments:
    parameters:
      - name: source
      - name: target
  entrypoint: main
  templates:
    - name: main
      dag:
        tasks:
          - name: extract
            template: extract-step
            arguments: {parameters: [{name: source, value: "{{workflow.parameters.source}}"}]}
          - name: load
            template: load-step
            dependencies: [extract]
            arguments: {parameters: [{name: target, value: "{{workflow.parameters.target}}"}]}

Multiple teams reuse via workflowTemplateRef.

Rule of thumb. Templates = DRY workflows.

Worked example — end-to-end GitOps deploy

1. Team commits new workflow YAML to git.
2. Argo CD detects change; syncs to team namespace.
3. Workflow appears in Argo UI.
4. Team submits or CronWorkflow triggers.
5. Metrics + logs stream to Prometheus + Loki.

Rule of thumb. Git → Argo CD → cluster → Argo Workflows.

argo workflows interview question on multi-tenant setup

A senior interviewer asks: "Design multi-tenant Argo Workflows for 5 teams on shared K8s cluster."

Solution Using namespace-per-team + RBAC + ResourceQuota + shared artifact repo

Complete solution:

• Namespace per team.
• ServiceAccount + Role + RoleBinding per team.
• ResourceQuota capping CPU/memory/pods.
• NetworkPolicy isolating traffic.
• ArtifactRepository per team.
• Argo CD ApplicationSet syncing from teams/{team}/workflows in git.
• Prometheus / Loki centralised observability.

Why this works — concept by concept:

• Namespace isolation — K8s primitive; RBAC + Quota enforced.
• ArtifactRepository per team — separate S3 buckets.
• ResourceQuota — prevents starvation.
• ApplicationSet — DRY multi-team GitOps.
• Observability shared — centralised metrics + logs.

SQL
Topic — SQL
SQL practice library

Practice →

SQL
Topic — optimization
SQL optimization drills

Practice →

---

Cheat sheet — Argo recipe list

• Workflow CRD — YAML declarative DAG.
• dag: or steps: template.
• entrypoint: names starting template.
• Parameters + artifacts + retry per step.
• activeDeadlineSeconds for timeout.
• ArtifactRepository — S3 / GCS for step outputs.
• EventSource + Sensor for triggers.
• CronWorkflow for schedules.
• WorkflowTemplate for reuse.
• ScheduledSparkApplication for Spark.
• argo submit / list / logs / delete.
• argo lint for validation.
• withParam for parallel loops.
• Argo CD for GitOps.
• ApplicationSet for multi-team.
• Namespace-per-env or per-team.
• ResourceQuota per namespace.
• NetworkPolicy isolation.
• Prometheus / Alertmanager for monitoring.
• SOPS or external-secrets for secrets.

Frequently asked questions

Argo Workflows or Airflow?

Depends on team stack. Argo Workflows wins for K8s-first shops with containerised workloads and platform teams comfortable in YAML. Airflow wins for Python-first DE teams with mature provider ecosystem needs (BigQuery, Snowflake, Slack, all the operators). Both are production-grade. Rule — if your team already writes YAML for K8s all day, Argo feels natural; if your team writes Python DAGs, Airflow.

How do I pass data between tasks?

Small values via parameters — {{steps.extract.outputs.parameters.count}}. Large data via artifacts — stored in S3/GCS via ArtifactRepository config; Argo handles upload/download automatically. Never rely on shared volume for correctness; use artifacts. For very large data (> 100 GB), consider passing S3 paths as parameters instead of full artifact upload.

How does Argo scale?

Each Workflow spawns pods on-demand via the Argo controller. Controller replicas scale (workflow-controller replicas: N). For high pod-churn, tune API server request limits. K8s API server becomes bottleneck at very high workflow throughput (thousands per minute). Rule — Argo scales well to hundreds of workflows/min per controller; beyond that, shard by namespace with dedicated controllers.

What's the difference between CronWorkflow and K8s CronJob?

CronWorkflow is Argo-specific — schedules Workflow CRDs and integrates with Argo's DAG features (artifacts, parameters, retry policies). K8s CronJob schedules Pods directly and doesn't do DAGs. Use CronWorkflow for multi-task workflows on a schedule; CronJob for simple single-container periodic tasks.

How do I test Argo Workflows locally?

argo lint for static validation of YAML. Run against a local kind (Kubernetes-in-Docker) cluster with Argo installed. For unit testing templates, use argo template validate. Some teams generate WorkflowTemplates programmatically from Python (via kubernetes client or hera-workflows library) and test with pytest. Rule — lint + kind + smoke test in CI.

How do I secure multi-tenant Argo?

Namespaces per tenant with RBAC restricting argoproj.io/v1alpha1 resources per namespace; ResourceQuotas capping pods/CPU/memory; NetworkPolicies isolating traffic; separate ArtifactRepositories per team; Argo has SSO integration for the UI (OIDC, SAML). Rule — treat Argo namespace like any multi-tenant K8s namespace; apply standard isolation.

Practice on PipeCode

• Drill the SQL practice library → — 450+ DE-focused questions covering pipeline orchestration.
• Sharpen SQL optimization drills → — DAG task tuning.
• Layer SQL join drills → — task chains often join across sources.
• For the broader SQL interview surface, take the SQL for Data Engineering course →.

Pipecode.ai is Leetcode for Data Engineering — every `argo workflows` pattern above ships with hands-on practice rooms where you write a 5-step DAG as Argo Workflow YAML, wire S3-arrival to trigger via EventSource + Sensor, factor a WorkflowTemplate for team-wide reuse, schedule via CronWorkflow instead of Airflow, and finally isolate multi-tenant namespaces with RBAC + ResourceQuota — the exact K8s-native orchestration fluency that senior platform interviews probe.

Practice SQL now →
Optimization drills →