DEV Community

Cover image for Helm Charts for Data Workloads: Airflow, Spark Operator, Trino, Postgres
Gowtham Potureddi
Gowtham Potureddi

Posted on

Helm Charts for Data Workloads: Airflow, Spark Operator, Trino, Postgres

#python #sql #interview #dataengineering

helm chart airflow is the deployment vehicle every DE platform team eventually adopts for data workloads on Kubernetes — the official Airflow chart with KubernetesExecutor or CeleryExecutor + KEDA autoscaling, the Spark Operator CRDs that turn Spark jobs into K8s-native resources, Trino for interactive SQL over a data lake, Postgres StatefulSets for the control-plane database. Every DE eventually installs a Helm chart; knowing how to override values.yaml per environment, how to choose an executor, how to secure secrets via K8s Secrets rather than baking into values, and when to reach for helmfile or Argo CD for GitOps is what separates a senior platform operator from a mid-level one.

The tour walks the five pillars — (1) the official Airflow Helm chart with executor choice (KubernetesExecutor vs CeleryExecutor), DAG sync via git-sync sidecar, KEDA-based worker autoscaling, (2) Spark Operator for K8s-native Spark with SparkApplication CRDs + dynamic allocation, (3) Trino + Postgres deployment with coordinator/workers, StatefulSet + PVC + backups (CloudNativePG for HA), (4) values.yaml override patterns for per-environment configuration + helmfile for multi-chart orchestration, and (5) Argo CD GitOps sync for declarative reconciliation from a git-of-truth repo. Every section ships a Solution-Tail interview answer — code, trace, output, why-this-works with __concept__ underlines.

PipeCode blog header for Helm Charts for Data Workloads — bold white headline 'HELM CHARTS' with subtitle 'Airflow · Spark · Trino · Postgres' and a stylised Helm-boat wheel with 4 K8s pod-cards around it on a dark gradient with pipecode.ai attribution.

Practice on SQL library →, SQL optimization drills →, and SQL join drills →.

On this page

1. Why Helm matters for DE in 2026

The helm chart mental model — templated K8s manifests, values-driven config, environment isolation

The one-sentence invariant: Helm is Kubernetes' de-facto package manager — a chart is a bundle of templated YAML manifests + a values.yaml schema; helm install release chart -f values.yaml renders the templates against your values and applies the resulting manifests to a cluster; upgrades preserve state via releases; every mainstream data workload (Airflow, Spark Operator, Trino, Postgres, Kafka, Superset, Metabase, Grafana) ships an official or community-maintained chart, so DE platform teams rarely write raw K8s manifests — they configure charts and let Helm do the templating.

Where Helm shows up in DE.

  • Self-hosted Airflow / Prefect / Dagster on K8s.
  • Spark Operator for job-per-CRD execution.
  • Trino / Presto for interactive SQL over data lake.
  • Postgres / MySQL for control-plane state (or CloudNativePG operator).
  • Kafka + Zookeeper / Kraft via Strimzi / Confluent operator + Helm.
  • Grafana / Prometheus for observability.
  • Metabase / Superset for BI.
  • Vault / cert-manager / cluster-autoscaler — infrastructure charts.

The Helm workflow.

# Add repo
helm repo add apache-airflow https://airflow.apache.org
helm repo update

# Install (or upgrade)
helm upgrade --install airflow apache-airflow/airflow \
    --namespace airflow --create-namespace \
    --version 1.15.0 \
    -f values-prod.yaml

# Inspect
helm list -n airflow
helm status airflow -n airflow
helm get values airflow -n airflow

# Rollback
helm rollback airflow 1 -n airflow

# Uninstall
helm uninstall airflow -n airflow
Enter fullscreen mode Exit fullscreen mode

What senior interviewers actually probe.

  • Executor choice. KubernetesExecutor vs CeleryExecutor tradeoffs.
  • DAG sync. Baked-in-image vs git-sync sidecar vs S3 sync.
  • Autoscaling. KEDA vs HPA vs manual.
  • Secrets. K8s Secrets vs Vault vs external secrets operator.
  • StatefulSet vs Deployment. Postgres uses StatefulSet.
  • PVC. Storage classes + backup strategies.
  • values.yaml layering. base + env-specific override.
  • helmfile. Multi-chart orchestration.
  • Argo CD. GitOps reconciliation.
  • Chart versioning. Pin chart version to avoid surprise breakage.

The five properties of a production Helm setup.

  • Env-per-values. values-dev.yaml, values-staging.yaml, values-prod.yaml.
  • Base + override layering. -f values-base.yaml -f values-prod.yaml.
  • Chart version pinning. --version 1.15.0 — never latest.
  • Secrets from K8s Secrets. Referenced in values.yaml as env vars.
  • GitOps. Argo CD watches git; syncs values.yaml on merge.

Worked example — the manual kubectl vs helm upgrade comparison

Detailed explanation. Before Helm: apply 40+ YAML files, remember dependencies, track version drift. With Helm: one command, one release, one rollback.

Question. Show both approaches.

Code — manual.

kubectl apply -f airflow-namespace.yaml
kubectl apply -f airflow-configmap.yaml
kubectl apply -f airflow-secrets.yaml
kubectl apply -f airflow-scheduler.yaml
kubectl apply -f airflow-webserver.yaml
kubectl apply -f airflow-workers.yaml
kubectl apply -f airflow-postgres.yaml
# ... 30 more files
Enter fullscreen mode Exit fullscreen mode

Code — Helm.

helm upgrade --install airflow apache-airflow/airflow \
    --namespace airflow --create-namespace \
    --version 1.15.0 \
    -f values-prod.yaml
Enter fullscreen mode Exit fullscreen mode

Step-by-step explanation.

  1. Manual: apply YAMLs; keep track of what's applied; upgrades = apply new YAMLs.
  2. Helm: one command; templating from values; release state tracked in K8s.
  3. Rollback: helm rollback airflow 1 reverts to previous release.
  4. Diff: helm diff upgrade (plugin) previews changes before apply.
  5. Values layering: base + env-specific overrides.

Output.

Aspect Manual Helm
Command count 40+ 1
Version tracking Manual Automatic
Rollback Reverse kubectl helm rollback
Env isolation Copy-paste YAML Values files
Team learning Everyone needs full K8s Values.yaml only

Rule of thumb. Every K8s data workload starts with Helm unless there's a strong reason not to.

Worked example — layered values files

Code.

# values-base.yaml
executor: KubernetesExecutor
config:
  loadExamples: false

# values-prod.yaml
webserver:
  replicas: 3
  resources:
    limits:
      cpu: 2
      memory: 4Gi
scheduler:
  replicas: 2
workers:
  replicas: 5
  keda:
    enabled: true
    maxReplicaCount: 20

# values-dev.yaml
webserver:
  replicas: 1
scheduler:
  replicas: 1
workers:
  replicas: 1
Enter fullscreen mode Exit fullscreen mode 
# Prod install
helm upgrade --install airflow apache-airflow/airflow \
    -f values-base.yaml \
    -f values-prod.yaml \
    -n airflow

# Dev install
helm upgrade --install airflow apache-airflow/airflow \
    -f values-base.yaml \
    -f values-dev.yaml \
    -n airflow-dev
Enter fullscreen mode Exit fullscreen mode

Rule of thumb. Base file for shared config; env file for scale/replica differences.

Worked example — dry-run + diff before apply

Code.

# Preview what would change
helm upgrade --install airflow apache-airflow/airflow \
    -f values-prod.yaml \
    --dry-run --debug \
    -n airflow

# Or with helm-diff plugin (better)
helm plugin install https://github.com/databus23/helm-diff
helm diff upgrade airflow apache-airflow/airflow -f values-prod.yaml -n airflow
Enter fullscreen mode Exit fullscreen mode

Rule of thumb. Always dry-run before apply on production releases.

Common beginner mistakes

  • Using latest chart version — surprise breaking upgrades.
  • Baking secrets in values.yaml — visible in git.
  • Not pinning version — non-reproducible.
  • Missing namespace — everything in default.
  • No values layering — copy-paste config per env.
  • No dry-run/diff — surprised by changes.

helm chart interview question on values layering strategy

A senior interviewer often opens with: "Design the values.yaml strategy for a Helm-based data platform with dev, staging, prod. What's the layering?"

Solution Using base + env-specific + secrets-external + helmfile 

repo/
├── charts/
│   ├── airflow/
│   │   ├── values-base.yaml
│   │   ├── values-dev.yaml
│   │   ├── values-staging.yaml
│   │   └── values-prod.yaml
│   ├── postgres/
│   │   └── values-*.yaml
│   └── ...
├── helmfile.yaml
└── .gitignore    # excludes secrets/
Enter fullscreen mode Exit fullscreen mode 
# helmfile.yaml
releases:
  - name: airflow
    chart: apache-airflow/airflow
    version: 1.15.0
    namespace: airflow
    values:
      - charts/airflow/values-base.yaml
      - charts/airflow/values-{{ .Environment.Name }}.yaml
    secrets:
      - charts/airflow/secrets.yaml.enc   # SOPS-encrypted
Enter fullscreen mode Exit fullscreen mode

Step-by-step trace.

Layer Purpose
values-base.yaml shared config (executor, image tags)
values-{env}.yaml env-specific (replicas, resources)
secrets.yaml.enc SOPS-encrypted secrets
helmfile.yaml orchestrates all charts

Output: Reproducible deploy across envs.

Why this works — concept by concept:

  • Base + env layering — no config duplication.
  • SOPS for secrets — encrypted-at-rest in git; decrypted at deploy.
  • helmfile for orchestration — multi-chart declarative deploy.
  • Chart version pin — reproducible.
  • GitOps possible — Argo CD watches git; syncs.

SQL
Topic — SQL
SQL practice library

Practice →

SQL Topic — optimization SQL optimization drills

Practice →

2. Airflow Helm chart

Executor choice + DAG sync + KEDA autoscaling — the three levers that define an Airflow deploy

The mental model in one line: the official apache/airflow Helm chart bundles the scheduler, webserver, workers, triggerer, and metadata Postgres into one release; three configuration decisions define production Airflow — the executor (KubernetesExecutor for isolated task pods with 5-15 sec startup vs CeleryExecutor for long-running workers with 100 ms task startup), the DAG-sync strategy (baked-in-image for immutable deploys vs git-sync sidecar for GitOps), and the worker autoscaling (KEDA scales workers based on pending task queue length; HPA scales on CPU/memory).

Visual diagram of Airflow Helm chart — executor selector card with 3 options KubernetesExecutor CeleryExecutor LocalExecutor radio glyphs, DAG sync card with git-sync sidecar glyph, workers vs scheduler card with a scheduler pod plus 3 worker pods and autoscale arrow; on a light PipeCode card.

Slot 1 — install the chart.

helm repo add apache-airflow https://airflow.apache.org
helm repo update
helm upgrade --install airflow apache-airflow/airflow \
    --namespace airflow --create-namespace \
    --version 1.15.0 \
    -f values-prod.yaml
Enter fullscreen mode Exit fullscreen mode

Slot 2 — executor choice.

  • KubernetesExecutor — one pod per task; best isolation; 5-15 sec startup overhead; unlimited scale.
  • CeleryExecutor — long-running worker pods pick up tasks from Redis/RabbitMQ; 100 ms task startup; scale = worker count.
  • KubernetesCeleryExecutor — hybrid; celery for many small tasks, k8s for heavy tasks.
  • LocalExecutor — dev only.
  • CeleryKubernetesExecutor — same as KubernetesCeleryExecutor. 
executor: KubernetesExecutor
Enter fullscreen mode Exit fullscreen mode

Slot 3 — DAG sync via git-sync.

dags:
  gitSync:
    enabled: true
    repo: git@github.com:myco/airflow-dags.git
    branch: main
    subPath: dags
    interval: 60
    depth: 1
    knownHosts: |
      github.com ssh-rsa AAAAB3NzaC1yc2E...
    sshKeySecret: airflow-git-ssh-secret
Enter fullscreen mode Exit fullscreen mode
  • Sidecar container clones repo every 60 sec.
  • DAGs mount as read-only volume to scheduler + workers.
  • Updates propagate without image rebuild.

Slot 4 — KEDA autoscaling.

workers:
  keda:
    enabled: true
    minReplicaCount: 1
    maxReplicaCount: 20
    pollingInterval: 5
    cooldownPeriod: 30
Enter fullscreen mode Exit fullscreen mode

KEDA monitors the Airflow metadata DB for pending task count; scales workers up/down.

Slot 5 — resource limits.

workers:
  resources:
    requests:
      cpu: 500m
      memory: 1Gi
    limits:
      cpu: 2
      memory: 4Gi

scheduler:
  resources:
    requests:
      cpu: 1
      memory: 2Gi
    limits:
      cpu: 4
      memory: 8Gi
Enter fullscreen mode Exit fullscreen mode

Never run workers without limits — one runaway task takes the node down.

Slot 6 — secrets.

extraEnv: |
  - name: AIRFLOW__CORE__FERNET_KEY
    valueFrom:
      secretKeyRef:
        name: airflow-secrets
        key: fernet-key
  - name: AIRFLOW__WEBSERVER__SECRET_KEY
    valueFrom:
      secretKeyRef:
        name: airflow-secrets
        key: webserver-secret-key
Enter fullscreen mode Exit fullscreen mode

Never in values.yaml directly.

Slot 7 — Postgres metadata.

postgresql:
  enabled: true
  auth:
    password: "{{ .Values.postgresql.auth.password }}"
    postgresPassword: "{{ .Values.postgresql.auth.postgresPassword }}"
  primary:
    persistence:
      size: 100Gi
Enter fullscreen mode Exit fullscreen mode

Or use external Postgres for production:

postgresql:
  enabled: false
data:
  metadataConnection:
    user: airflow
    pass: ""   # from secret
    host: external-postgres.example.com
    port: 5432
    db: airflow_meta
Enter fullscreen mode Exit fullscreen mode

Slot 8 — RBAC.

rbac:
  create: true
  createSCCRoleBinding: false
Enter fullscreen mode Exit fullscreen mode

Chart creates ServiceAccount + Role + RoleBinding for the executor.

Slot 9 — ingress.

webserver:
  ingress:
    enabled: true
    hosts:
      - airflow.example.com
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
    tls:
      - hosts:
          - airflow.example.com
        secretName: airflow-tls
Enter fullscreen mode Exit fullscreen mode

Slot 10 — logging.

logs:
  persistence:
    enabled: true
    size: 50Gi
Enter fullscreen mode Exit fullscreen mode

Or remote logging to S3:

config:
  logging:
    remote_logging: 'True'
    remote_base_log_folder: 's3://mybucket/airflow-logs'
    remote_log_conn_id: 's3_default'
Enter fullscreen mode Exit fullscreen mode

Common beginner mistakes

  • Using LocalExecutor in prod.
  • No resource limits.
  • Baking secrets in values.yaml.
  • git-sync without knownHosts.
  • Missing KEDA — scaling manual.
  • No remote logging — logs lost on pod restart.

Worked example — production-ready Airflow values.yaml

Code.

# values-prod.yaml
executor: KubernetesExecutor

images:
  airflow:
    repository: myco/airflow
    tag: "2.10.0-custom"

config:
  core:
    load_examples: 'False'
    max_active_runs_per_dag: 3
  logging:
    remote_logging: 'True'
    remote_base_log_folder: 's3://myco-airflow-logs'

webserver:
  replicas: 2
  resources:
    limits: {cpu: 2, memory: 4Gi}
  ingress:
    enabled: true
    hosts: [{name: airflow.myco.com}]

scheduler:
  replicas: 2
  resources:
    limits: {cpu: 4, memory: 8Gi}

workers:
  keda:
    enabled: true
    minReplicaCount: 1
    maxReplicaCount: 20
  resources:
    limits: {cpu: 2, memory: 4Gi}

dags:
  gitSync:
    enabled: true
    repo: git@github.com:myco/airflow-dags.git
    branch: main
    subPath: dags
    interval: 60

postgresql:
  enabled: false
data:
  metadataConnection:
    host: postgres-airflow.myco.svc.cluster.local
Enter fullscreen mode Exit fullscreen mode

Rule of thumb. Production Airflow requires KubernetesExecutor + KEDA + git-sync + external Postgres + S3 logs.

Worked example — KEDA autoscaling in action

Code / diagram.

Baseline: 1 worker
Task queue: 0 pending
Time = t0

t0+30s: 100 DAGs trigger → 500 tasks queued
KEDA polls metadata DB every 5s → sees 500 pending
KEDA target: maxReplicaCount 20
Scale up: 1 → 5 → 10 → 20 workers over 30 sec

t0+3min: tasks running; queue draining
t0+8min: queue = 0
Cooldown 30 sec
Scale down: 20 → 10 → 5 → 1
Enter fullscreen mode Exit fullscreen mode

Rule of thumb. KEDA fastens the "worker starts when needed" loop.

Worked example — DAG sync verification

Code.

# Verify DAGs synced
kubectl exec -it airflow-scheduler-0 -n airflow -- ls /opt/airflow/dags/
# See DAG files from git

# Check sync interval
kubectl logs -f airflow-scheduler-0 -n airflow -c git-sync
# Sync every 60 sec
Enter fullscreen mode Exit fullscreen mode

Rule of thumb. git-sync sidecar puts DAGs in shared volume; both scheduler and workers mount it.

helm chart airflow interview question on executor choice

A senior interviewer asks: "You have 5000 tasks/hour with heavy compute (memory-intensive Python transforms). KubernetesExecutor or CeleryExecutor?"

Solution Using KubernetesCeleryExecutor for hybrid 

executor: CeleryKubernetesExecutor

workers:
  replicas: 5   # long-running celery workers for small tasks
  keda:
    enabled: true
    maxReplicaCount: 20

# In DAG code, mark heavy tasks:
@task(queue="kubernetes", executor_config={"pod_override": ...})
def heavy_transform(...):
    ...
Enter fullscreen mode Exit fullscreen mode

Why this works — concept by concept:

  • CeleryExecutor for small tasks — 100 ms startup; workers pre-warmed.
  • KubernetesExecutor for heavy tasks — per-task pod with big memory.
  • Hybrid — best of both worlds.
  • KEDA scales celery workers — matches load.

SQL
Topic — SQL
SQL practice library

Practice →

SQL Topic — optimization SQL optimization drills

Practice →

3. Spark Operator on K8s

SparkApplication CRD — declarative Spark jobs

The mental model in one line: the Spark Operator (spark-operator Helm chart) installs a K8s controller + a SparkApplication Custom Resource Definition; instead of spark-submit, you kubectl apply a SparkApplication YAML which the operator translates into driver + executor pods, monitors lifecycle, restarts on failure, and integrates cleanly with cluster autoscaler; dynamic allocation scales executors based on pending tasks; logs go to S3/GCS for post-mortem analysis.

Visual diagram of Spark Operator — SparkApplication CRD card with YAML strip, driver pod glyph plus 5 executor pods spawning, dynamic allocation card with scale-up/down glyph min/max chips; on a light PipeCode card.

Slot 1 — install operator.

helm repo add spark-operator https://kubeflow.github.io/spark-operator
helm upgrade --install spark-operator spark-operator/spark-operator \
    --namespace spark --create-namespace \
    --set enableWebhook=true
Enter fullscreen mode Exit fullscreen mode

Slot 2 — SparkApplication CRD.

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: daily-etl
  namespace: spark
spec:
  type: Python
  mode: cluster
  image: myco/spark:3.5.1
  imagePullPolicy: Always
  mainApplicationFile: local:///opt/spark/work-dir/etl.py
  sparkVersion: 3.5.1
  restartPolicy:
    type: OnFailure
    onFailureRetries: 3
    onFailureRetryInterval: 10
  driver:
    cores: 1
    coreLimit: "1200m"
    memory: 2g
    serviceAccount: spark
    labels:
      app: etl
  executor:
    cores: 2
    instances: 5
    memory: 4g
    labels:
      app: etl
  dynamicAllocation:
    enabled: true
    initialExecutors: 2
    minExecutors: 1
    maxExecutors: 20
  sparkConf:
    "spark.eventLog.enabled": "true"
    "spark.eventLog.dir": "s3a://myco-spark-history/logs"
    "spark.hadoop.fs.s3a.aws.credentials.provider": "com.amazonaws.auth.EnvironmentVariableCredentialsProvider"
Enter fullscreen mode Exit fullscreen mode

Slot 3 — dynamic allocation.

  • Scales executor count based on pending tasks.
  • initialExecutors, minExecutors, maxExecutors.
  • Requires shuffle service or external shuffle (K8s uses PVC-backed).
  • Great for variable workload; overhead is modest.

Slot 4 — resources + limits.

  • driver.cores / driver.memory — driver pod.
  • executor.cores / executor.memory — per-executor pod.
  • Set coreLimit explicitly to prevent CPU throttling surprises.

Slot 5 — service accounts + RBAC.

Spark operator creates a spark service account with permissions to spawn driver/executor pods.

driver:
  serviceAccount: spark
Enter fullscreen mode Exit fullscreen mode

Slot 6 — logs.

  • Configure spark.eventLog.dir → S3/GCS.
  • Post-mortem via Spark History Server.
  • Or Loki + Grafana for streaming logs.

Slot 7 — Airflow integration.

from airflow.providers.cncf.kubernetes.operators.spark_kubernetes import SparkKubernetesOperator

etl = SparkKubernetesOperator(
    task_id="etl",
    namespace="spark",
    application_file="etl.yaml",   # SparkApplication CRD
    kubernetes_conn_id="kubernetes_default",
    do_xcom_push=False,
)
Enter fullscreen mode Exit fullscreen mode

Slot 8 — status watching.

kubectl get sparkapplication daily-etl -n spark
# NAME       STATUS     ATTEMPTS   START                  ...
# daily-etl  COMPLETED  1          2026-07-12T04:00:00Z   ...

kubectl describe sparkapplication daily-etl -n spark
Enter fullscreen mode Exit fullscreen mode

Slot 9 — restart policy.

  • Always — always restart.
  • OnFailure — restart only on non-zero exit.
  • Never — never restart.
  • onFailureRetries — number of retries.

Slot 10 — scheduled Spark via ScheduledSparkApplication.

apiVersion: sparkoperator.k8s.io/v1beta2
kind: ScheduledSparkApplication
metadata:
  name: daily-etl
spec:
  schedule: "0 4 * * *"   # cron: 4 AM daily
  concurrencyPolicy: Forbid
  template:
    # same as SparkApplication spec
Enter fullscreen mode Exit fullscreen mode

Common beginner mistakes

  • No serviceAccount configured — pod-spawn permission denied.
  • No dynamic allocation — over-provisioned executors.
  • Missing shuffle service — dynamic allocation limps.
  • Java version mismatch between operator + Spark image.
  • No event log dir — no history for debugging.

Worked example — end-to-end daily ETL

SparkApplication YAML + Airflow trigger.

# daily-etl.yaml
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: daily-etl-{{ ds }}
  namespace: spark
spec:
  type: Python
  mode: cluster
  image: myco/spark:3.5.1
  mainApplicationFile: s3a://myco/etl/etl.py
  sparkVersion: 3.5.1
  arguments:
    - --date
    - "{{ ds }}"
  driver: {cores: 1, memory: 2g, serviceAccount: spark}
  executor: {cores: 2, instances: 5, memory: 4g}
  dynamicAllocation: {enabled: true, minExecutors: 2, maxExecutors: 20}
Enter fullscreen mode Exit fullscreen mode 
# In Airflow DAG
from airflow.providers.cncf.kubernetes.operators.spark_kubernetes import SparkKubernetesOperator

etl = SparkKubernetesOperator(
    task_id="daily_etl",
    namespace="spark",
    application_file="daily-etl.yaml",
    do_xcom_push=False,
    dag=dag,
)
Enter fullscreen mode Exit fullscreen mode

Rule of thumb. Airflow schedules; Spark Operator runs.

Worked example — dynamic allocation in action

Code / diagram.

Job starts: 2 executors (initialExecutors)
Task queue grows: 5000 tasks pending
Spark asks for more: scale to 10
Cluster autoscaler wakes: adds nodes
More tasks queue: scale to 20 (max)
Tasks drain: scale down to 5
Job complete: scale to 0
Enter fullscreen mode Exit fullscreen mode

Rule of thumb. Dynamic allocation + cluster autoscaler = elastic Spark.

Worked example — post-mortem via Spark History Server

Code.

# Install Spark History Server via Helm
helm repo add spark-history-server ...
helm install spark-history spark-history-server/spark-history-server \
    --set eventLogDir=s3a://myco-spark-history/logs
Enter fullscreen mode Exit fullscreen mode

Access at spark-history.myco.com; view completed jobs.

Rule of thumb. Always enable event logging; Spark History Server for post-mortem.

spark operator interview question on dynamic allocation vs static

A senior interviewer asks: "Your Spark jobs have highly variable load — some hours 1 executor is enough, other hours you need 20. Design."

Solution Using dynamic allocation + cluster autoscaler + node pool 

# SparkApplication
spec:
  dynamicAllocation:
    enabled: true
    initialExecutors: 1
    minExecutors: 1
    maxExecutors: 20
  sparkConf:
    "spark.dynamicAllocation.executorIdleTimeout": "60s"
    "spark.dynamicAllocation.schedulerBacklogTimeout": "5s"

# Node pool with cluster autoscaler
# spark-nodes: min=0, max=20 nodes
Enter fullscreen mode Exit fullscreen mode

Why this works — concept by concept:

  • Dynamic allocation — Spark asks for executors as needed.
  • Cluster autoscaler — K8s provisions nodes when pods pending.
  • Idle timeout — release executors after 60 sec idle.
  • Backlog timeout — request executors within 5 sec of task queue.
  • Cost — pay only for what you use.

SQL
Topic — optimization
SQL optimization drills

Practice →

SQL Topic — joins SQL join drills

Practice →

4. Trino + Postgres helm

Stateless Trino + StatefulSet Postgres — the two most common data control-plane deploys

The mental model in one line: Trino is stateless — one coordinator + N workers reading from a data lake, scaled via Helm replica count; Postgres is stateful — StatefulSet + PersistentVolumeClaim for data survival across pod restarts + a backup schedule; for HA Postgres, use the CloudNativePG operator instead of raw StatefulSet, which handles automatic failover, backups, and PITR.

Visual diagram of Trino + Postgres helm — Trino card with coordinator pod plus 5 worker pods sharing a catalog, Postgres StatefulSet card with 3 replica pods with PVC glyphs and backup schedule chip; on a light PipeCode card.

Slot 1 — Trino Helm.

helm repo add trino https://trinodb.github.io/charts
helm upgrade --install trino trino/trino \
    -f trino-values.yaml \
    -n trino --create-namespace
Enter fullscreen mode Exit fullscreen mode

Slot 2 — Trino values.yaml.

coordinator:
  jvm:
    maxHeapSize: "8G"
  resources:
    requests: {cpu: 2, memory: 8Gi}
    limits: {cpu: 4, memory: 16Gi}

worker:
  replicas: 5
  jvm:
    maxHeapSize: "16G"
  resources:
    requests: {cpu: 4, memory: 16Gi}
    limits: {cpu: 8, memory: 32Gi}

catalogs:
  hive: |
    connector.name=hive
    hive.metastore.uri=thrift://hive-metastore:9083
    hive.s3.aws-access-key=${ENV:AWS_ACCESS_KEY_ID}
    hive.s3.aws-secret-key=${ENV:AWS_SECRET_ACCESS_KEY}

  iceberg: |
    connector.name=iceberg
    iceberg.catalog.type=hive
    iceberg.catalog.uri=thrift://hive-metastore:9083

  postgres: |
    connector.name=postgresql
    connection-url=jdbc:postgresql://postgres:5432/db
    connection-user=trino
    connection-password=${ENV:POSTGRES_PASSWORD}
Enter fullscreen mode Exit fullscreen mode

Slot 3 — Trino architecture.

  • Coordinator — parses SQL, plans, distributes.
  • Workers — execute stages.
  • Catalogs — Hive, Iceberg, Postgres, Kafka, Elasticsearch, MongoDB, more.
  • Stateless — kill and replace workers; queries retry.

Slot 4 — Bitnami Postgres Helm.

helm repo add bitnami https://charts.bitnami.com/bitnami
helm upgrade --install postgres bitnami/postgresql \
    --namespace postgres --create-namespace \
    -f postgres-values.yaml
Enter fullscreen mode Exit fullscreen mode

Slot 5 — Postgres values.yaml.

auth:
  postgresPassword: "{{ ...from secret... }}"
  username: airflow
  password: "{{ ... }}"
  database: airflow_meta

primary:
  persistence:
    enabled: true
    size: 100Gi
    storageClass: gp3

  resources:
    requests: {cpu: 2, memory: 4Gi}
    limits: {cpu: 4, memory: 8Gi}

  extendedConfiguration: |
    max_connections = 200
    shared_buffers = 2GB
    effective_cache_size = 6GB
    work_mem = 32MB

backup:
  enabled: true
  cronjob:
    schedule: "0 3 * * *"
    storage:
      size: 100Gi

metrics:
  enabled: true
  serviceMonitor:
    enabled: true
Enter fullscreen mode Exit fullscreen mode

Slot 6 — CloudNativePG for production HA.

helm install cnpg cloudnative-pg/cloudnative-pg -n cnpg-system

# Create Postgres cluster CRD
kubectl apply -f - <<EOF
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: airflow-postgres
  namespace: airflow
spec:
  instances: 3
  primaryUpdateStrategy: unsupervised
  storage:
    size: 100Gi
    storageClass: gp3
  backup:
    barmanObjectStore:
      destinationPath: s3://myco-pg-backups
    retentionPolicy: "30d"
EOF
Enter fullscreen mode Exit fullscreen mode

CloudNativePG handles failover, backups, PITR.

Slot 7 — Trino query flow.

Client → coordinator (localhost:8080)
Coordinator → parse SQL → plan → dispatch stages
Worker 1 → scan hive catalog → produce rows
Worker 2 → scan hive catalog → produce rows
Worker 3 → shuffle exchange
...
Coordinator ← receive final result → return to client
Enter fullscreen mode Exit fullscreen mode

Slot 8 — Trino authentication.

server:
  config:
    http:
      authenticationType: "password"
  authentication:
    password:
      passwordAuthenticator:
        password-authenticator.name: file
        file.password-file: /etc/trino/password.db
Enter fullscreen mode Exit fullscreen mode

Slot 9 — Trino resource groups.

# Manage query prioritisation
resourceGroups:
  bi: {maxRunning: 20, maxQueued: 100}
  etl: {maxRunning: 5, maxQueued: 50}
Enter fullscreen mode Exit fullscreen mode

Slot 10 — Trino ingress.

ingress:
  enabled: true
  hosts: [{name: trino.myco.com}]
Enter fullscreen mode Exit fullscreen mode

Common beginner mistakes

  • Postgres as Deployment instead of StatefulSet — data lost on pod restart.
  • No PVC — data lost.
  • No backups — irrecoverable.
  • Trino coordinator too small.
  • No catalog config — nothing to query.

Worked example — daily Trino query from Airflow

Code.

from airflow.providers.trino.operators.trino import TrinoOperator

query_daily = TrinoOperator(
    task_id="query_daily",
    trino_conn_id="trino_default",
    sql="""
        INSERT INTO iceberg.analytics.daily_metrics
        SELECT
            date_trunc('day', event_ts) AS day,
            COUNT(*) AS events
        FROM iceberg.raw.events
        WHERE event_ts >= current_date - interval '1' day
        GROUP BY 1
    """,
)
Enter fullscreen mode Exit fullscreen mode

Rule of thumb. Airflow schedules; Trino queries.

Worked example — CloudNativePG for HA Postgres

Code.

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: airflow-postgres
spec:
  instances: 3
  storage:
    size: 100Gi
  postgresql:
    parameters:
      max_connections: "200"
      shared_buffers: "2GB"
  backup:
    barmanObjectStore:
      destinationPath: s3://myco-pg-backups
    retentionPolicy: "30d"
  managed:
    roles:
      - name: airflow
        ensure: present
        passwordSecret:
          name: airflow-pg-secret
Enter fullscreen mode Exit fullscreen mode

Rule of thumb. For prod Postgres, CloudNativePG > raw StatefulSet.

Worked example — Trino catalog for Iceberg

Code.

catalogs:
  iceberg: |
    connector.name=iceberg
    iceberg.catalog.type=rest
    iceberg.rest-catalog.uri=http://iceberg-rest:8181
    iceberg.rest-catalog.warehouse=s3://myco-iceberg
Enter fullscreen mode Exit fullscreen mode

Rule of thumb. Trino + Iceberg + REST catalog = modern data lake stack.

helm chart trino interview question on scaling

A senior interviewer asks: "Trino queries are slow. Workers CPU at 90%. Add more workers via Helm."

Solution Using scale replicas + HPA 

worker:
  replicas: 10                    # up from 5

  autoscaling:
    enabled: true
    minReplicas: 5
    maxReplicas: 20
    targetCPUUtilizationPercentage: 70
Enter fullscreen mode Exit fullscreen mode 
helm upgrade trino trino/trino -f values.yaml -n trino
Enter fullscreen mode Exit fullscreen mode

Why this works — concept by concept:

  • Trino stateless — safe to scale.
  • HPA on CPU — auto-scales up on load.
  • Query re-dispatch — new workers pick up new stages; existing queries drain on old workers.
  • Coordinator unchanged — only workers scaled.

SQL
Topic — SQL
SQL practice library

Practice →

SQL Topic — optimization SQL optimization drills

Practice →

5. values.yaml + GitOps

Layered values + helmfile + Argo CD — the reproducible platform deploy

The mental model in one line: a production Helm platform layers configuration as values-base.yaml (shared) + values-{env}.yaml (env-specific override) applied together via helm upgrade -f base -f env; multi-chart platforms use helmfile to declare the full chart set; Argo CD watches the git repo containing helmfile + values, reconciles the cluster to match git state, automates rollout, and provides visual sync status per release.

Visual diagram of values.yaml + GitOps — values.yaml scroll with three env overrides (dev/staging/prod), helmfile card orchestrating multiple charts, Argo CD card syncing from git to cluster with sync arrow; on a light PipeCode card.

Slot 1 — values layering command.

helm upgrade --install airflow apache-airflow/airflow \
    -f values-base.yaml \
    -f values-prod.yaml \
    -n airflow
Enter fullscreen mode Exit fullscreen mode

Later files override earlier.

Slot 2 — file structure.

platform-config/
├── charts/
│   ├── airflow/
│   │   ├── values-base.yaml
│   │   ├── values-dev.yaml
│   │   ├── values-staging.yaml
│   │   └── values-prod.yaml
│   ├── trino/
│   │   └── values-*.yaml
│   ├── postgres/
│   │   └── values-*.yaml
│   └── spark-operator/
│       └── values-*.yaml
├── helmfile.yaml
├── argocd-apps/
│   ├── airflow-app.yaml
│   └── ...
└── README.md
Enter fullscreen mode Exit fullscreen mode

Slot 3 — helmfile.yaml.

environments:
  dev:
    values: [{cluster: "dev-cluster"}]
  staging: {}
  prod:
    values: [{cluster: "prod-cluster"}]

repositories:
  - name: apache-airflow
    url: https://airflow.apache.org
  - name: trino
    url: https://trinodb.github.io/charts
  - name: bitnami
    url: https://charts.bitnami.com/bitnami

releases:
  - name: airflow
    chart: apache-airflow/airflow
    version: 1.15.0
    namespace: airflow
    values:
      - charts/airflow/values-base.yaml
      - charts/airflow/values-{{ .Environment.Name }}.yaml
    needs:
      - postgres/postgres

  - name: trino
    chart: trino/trino
    version: 0.29.0
    namespace: trino
    values:
      - charts/trino/values-{{ .Environment.Name }}.yaml

  - name: postgres
    chart: bitnami/postgresql
    version: 15.5.0
    namespace: postgres
    values:
      - charts/postgres/values-{{ .Environment.Name }}.yaml
Enter fullscreen mode Exit fullscreen mode

Slot 4 — helmfile commands.

# Preview
helmfile -e prod diff

# Apply
helmfile -e prod apply

# Sync a specific release
helmfile -e prod -l name=airflow apply
Enter fullscreen mode Exit fullscreen mode

Slot 5 — Argo CD Application CRD.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: airflow-prod
  namespace: argocd
spec:
  project: default
  source:
    repoURL: git@github.com:myco/platform-config.git
    targetRevision: main
    path: charts/airflow
    helm:
      valueFiles:
        - values-base.yaml
        - values-prod.yaml
      version: v3
  destination:
    server: https://kubernetes.default.svc
    namespace: airflow
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
Enter fullscreen mode Exit fullscreen mode

Slot 6 — Argo CD auto-sync.

  • Watches git; syncs on commit.
  • prune: true — deletes resources removed from git.
  • selfHeal: true — reverts manual kubectl changes.

Slot 7 — SOPS for secrets.

# Encrypt
sops --encrypt --pgp <fingerprint> secrets.yaml > secrets.enc.yaml

# In helmfile
releases:
  - name: airflow
    secrets:
      - charts/airflow/secrets.enc.yaml
Enter fullscreen mode Exit fullscreen mode

Helmfile decrypts at apply time; git only stores encrypted.

Slot 8 — external-secrets operator.

Alternative to SOPS:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: airflow-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: airflow-secrets
  data:
    - secretKey: fernet-key
      remoteRef:
        key: /airflow/fernet-key
Enter fullscreen mode Exit fullscreen mode

Slot 9 — chart version pinning.

releases:
  - name: airflow
    chart: apache-airflow/airflow
    version: 1.15.0    # exact pin
Enter fullscreen mode Exit fullscreen mode

Never latest. Bumping requires PR + review.

Slot 10 — the four-workload matrix.

Workload Chart Storage Autoscale
Airflow apache-airflow logs on PVC / S3 KEDA
Spark spark-operator S3 event logs dynamic allocation
Trino trinodb/charts none (stateless) HPA
Postgres bitnami/postgresql or CloudNativePG PVC + backups vertical only

Common beginner mistakes

  • No values layering — copy-paste per env.
  • Baking secrets in values.yaml.
  • Not pinning chart versions.
  • Argo CD sync without prune — orphan resources.
  • helmfile without needs — order-of-operations wrong.

Worked example — end-to-end platform via helmfile

Code.

# helmfile.yaml
environments:
  prod:
    values:
      - {cluster: "prod-us-east"}

releases:
  - name: cert-manager
    chart: jetstack/cert-manager
    namespace: cert-manager
    version: 1.13.0
    values:
      - installCRDs: true

  - name: postgres
    chart: bitnami/postgresql
    namespace: postgres
    version: 15.5.0
    values:
      - charts/postgres/values-{{ .Environment.Name }}.yaml
    needs: [cert-manager/cert-manager]

  - name: airflow
    chart: apache-airflow/airflow
    namespace: airflow
    version: 1.15.0
    values:
      - charts/airflow/values-base.yaml
      - charts/airflow/values-{{ .Environment.Name }}.yaml
    secrets:
      - charts/airflow/secrets-{{ .Environment.Name }}.enc.yaml
    needs: [postgres/postgres]

  - name: trino
    chart: trino/trino
    namespace: trino
    version: 0.29.0
    values:
      - charts/trino/values-{{ .Environment.Name }}.yaml

  - name: spark-operator
    chart: spark-operator/spark-operator
    namespace: spark
    version: 1.1.27
Enter fullscreen mode Exit fullscreen mode 
helmfile -e prod apply
Enter fullscreen mode Exit fullscreen mode

Rule of thumb. helmfile for multi-chart platforms.

Worked example — Argo CD app-of-apps pattern

Code.

# root Application that manages all others
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: platform
  namespace: argocd
spec:
  source:
    repoURL: git@github.com:myco/platform-config.git
    path: argocd-apps/
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated: {prune: true, selfHeal: true}

# Individual apps
argocd-apps/
├── airflow.yaml
├── trino.yaml
├── postgres.yaml
└── spark-operator.yaml
Enter fullscreen mode Exit fullscreen mode

Rule of thumb. App-of-apps for multi-chart GitOps.

Worked example — rolling back a bad release

Code.

# See history
helm history airflow -n airflow

# Rollback to revision 3
helm rollback airflow 3 -n airflow

# Or in Argo CD UI: click "Rollback" on Application
Enter fullscreen mode Exit fullscreen mode

Rule of thumb. Helm keeps release history; rollback is one command.

helm chart interview question on multi-cluster deploy

A senior interviewer asks: "Deploy the same platform to 5 K8s clusters (us-east, us-west, eu-west, ap-south, ap-northeast) with slight config differences. How?"

Solution Using helmfile environments + Argo CD ApplicationSet 

# helmfile.yaml
environments:
  us-east: {values: [{region: us-east-1}]}
  us-west: {values: [{region: us-west-2}]}
  eu-west: {values: [{region: eu-west-1}]}
  ap-south: {values: [{region: ap-south-1}]}
  ap-northeast: {values: [{region: ap-northeast-1}]}

releases:
  - name: airflow
    chart: apache-airflow/airflow
    values:
      - charts/airflow/values-base.yaml
      - charts/airflow/values-region-{{ .Values.region }}.yaml

# OR Argo CD ApplicationSet with cluster generator
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: airflow-all-regions
spec:
  generators:
    - clusters: {}
  template:
    metadata:
      name: airflow-{{ name }}
    spec:
      source:
        repoURL: ...
        path: charts/airflow
        helm:
          valueFiles:
            - values-base.yaml
            - values-{{ metadata.labels.region }}.yaml
      destination:
        server: '{{ server }}'
        namespace: airflow
Enter fullscreen mode Exit fullscreen mode

Why this works — concept by concept:

  • ApplicationSet with cluster generator — one template renders N applications.
  • Region label — selects the right values file.
  • helmfile alternative — script-based orchestration.
  • DRY — one config, N clusters.

SQL
Topic — SQL
SQL practice library

Practice →

SQL
Topic — optimization
SQL optimization drills

Practice →

Cheat sheet — Helm recipe list

  • helm repo add / update / install / upgrade / rollback / list / status — basics.
  • helm upgrade --install release chart -f values.yaml — idempotent apply.
  • --version 1.15.0 — always pin.
  • -n namespace --create-namespace — target namespace.
  • --dry-run --debug — preview.
  • helm-diff plugin — diff before apply.
  • Airflow — KubernetesExecutor + KEDA + git-sync + external Postgres.
  • Spark Operator — SparkApplication CRD + dynamic allocation + event logs.
  • Trino — coordinator + workers + catalog configs.
  • Postgres — StatefulSet + PVC + backups; CloudNativePG for HA.
  • Values layering — -f base -f env.
  • helmfile — multi-chart orchestration.
  • Argo CD — GitOps sync.
  • ApplicationSet — multi-cluster templating.
  • SOPS — encrypt secrets in git.
  • external-secrets — sync from cloud secret store.
  • HPA / KEDA — autoscaling.
  • HEALTHCHECK / readiness — K8s probes.
  • ResourceQuota — namespace-level caps.
  • NetworkPolicy — inter-pod isolation.
  • Never latest — always pin.

Frequently asked questions

KubernetesExecutor vs CeleryExecutor for Airflow?

KubernetesExecutor launches one pod per task — best isolation, per-task resource limits, but 5-15 second startup overhead per task. CeleryExecutor keeps worker pods running to pick up tasks — 100 ms task startup but you pay for idle workers. Use KubernetesExecutor for long-running heavy tasks (dbt, Spark, big Python), CeleryExecutor for many short tasks (data quality checks, notification tasks). Hybrid KubernetesCeleryExecutor combines both — celery for defaults, kubernetes for tasks that request it via executor_config. Rule of thumb — start with KubernetesExecutor for simplicity; add CeleryExecutor when task startup latency dominates.

Should I run Postgres in K8s?

For dev / staging / non-critical: sure — bitnami/postgresql chart with PVC is fine. For production HA with automatic failover, backups, and PITR, use the CloudNativePG operator which handles all of that natively. For very high-throughput or heavily-tuned workloads, prefer managed services (RDS, Cloud SQL, Azure Postgres) — the operational cost of running your own is high. Rule — control-plane state (Airflow metadata DB) can be self-hosted with CloudNativePG; user-data or heavy OLTP should be managed.

How does DAG sync work?

The git-sync sidecar container clones the git repo into a shared volume every N seconds (default 60). The scheduler and workers mount the volume; DAG changes propagate within the sync interval without image rebuild. Handles authentication via SSH key or HTTPS token via K8s Secret. Alternative: bake DAGs into the Airflow image for immutable deploys — image rebuild per DAG change but no runtime git dependency. Rule — git-sync for GitOps and rapid iteration; baked image for stable production or air-gapped deploys.

What's KEDA and why prefer it over HPA?

KEDA (Kubernetes Event-Driven Autoscaling) scales deployments based on external metrics — for Airflow, worker count scales based on pending task queue length via the Airflow metadata DB. HPA scales based on CPU/memory only, which for Airflow workers correlates poorly with load — a worker with pending tasks is not always CPU-bound. KEDA is more responsive and matches Airflow semantics better. Rule — KEDA for Airflow workers; HPA for stateless services like Trino workers where CPU/memory correlate with load.

Argo CD or Flux?

Both are solid GitOps operators. Argo CD has a polished UI, an Application CRD with clear semantics, and the ApplicationSet for multi-cluster templating. Flux is more modular, closer to plain Kubernetes semantics, and uses multiple CRDs (GitRepository, HelmRelease, Kustomization) that compose. Pick based on team preference — Argo CD wins on UX for platform teams; Flux wins on flexibility for advanced users. Both support Helm charts natively. Rule — Argo CD if you want a UI and streamlined app model; Flux if you want maximum flexibility.

How do I test Helm charts?

helm lint for static checks; helm template to render locally and inspect the manifests; helm install --dry-run to validate against the cluster's API server without actually creating resources; chart-testing (ct) for automated CI validation. For real integration, deploy to a kind (Kubernetes-in-Docker) cluster in CI and run smoke tests against the deployed release. helm-diff plugin for previewing changes before apply. Rule — every chart PR should pass lint + template + kind integration test in CI.

Practice on PipeCode

Pipecode.ai is Leetcode for Data Engineering — every `helm chart airflow` pattern above ships with hands-on practice rooms where you install the Airflow chart with KubernetesExecutor + KEDA + git-sync, deploy Spark jobs as SparkApplication CRDs via the Spark Operator, scale a Trino coordinator + workers cluster via HPA, run Postgres StatefulSet with backups via CloudNativePG, layer values-base + values-prod for env isolation, and wire the whole platform through Argo CD ApplicationSets for multi-cluster GitOps — the exact K8s data-platform fluency that senior DE and platform interviews probe.

Practice SQL now →
Optimization drills →

Top comments (0)