DEV Community: Chirag Bhatia

Your Spark job dies before Prometheus can even scrape it. How do you monitor dead containers without OOM crashing the Pushgateway? Read: https://dev.to/thestaffblueprint/the-green-tick-fallacy-why-batch-observability-is-fundamentally-different-398c

Chirag Bhatia — Mon, 18 May 2026 15:32:50 +0000

Chirag Bhatia for The Staff Blueprint

May 17

The Green Tick Fallacy — Why Batch Observability is Fundamentally Different

#observability #airflow #prometheus #grafana

Comments

4 min read

The Architecture — Prometheus, Grafana, and StatsD for Batch Workloads

Chirag Bhatia — Sun, 17 May 2026 12:19:19 +0000

(This is Part 2 of the Batch Workloads Observability series. Read Part 1: The Green Tick Fallacy first for context.)

The Default Choice: Prometheus

When engineering teams realise they need to extract internal application metrics from their batch pipelines (to escape the “Green Tick Fallacy”), they inevitably reach for Prometheus.

It makes perfect sense. Prometheus is the industry standard. Grafana integrates beautifully with it. PromQL is incredibly powerful. Your infrastructure team already has it running. (If you want a detailed breakdown of each component and how the standard Prometheus architecture works under the hood, check out this comprehensive guide by DevOpsCube.)

But there is a fundamental mismatch: Prometheus is a pull-based system. It expects to scrape a /metrics HTTP endpoint exposed by a continuously running service.

Batch jobs, however, are ephemeral. A Spark job might spin up, process terabytes of data in 45 seconds, and vanish. By the time the Prometheus scraper comes around (typically every 15 to 30 seconds), the container is already dead. You cannot scrape batch jobs—you must push telemetry out before the container dies.

The Push Problem & The Proxy

To solve the push-vs-pull mismatch, the standard architecture introduces the Prometheus Pushgateway.

The idea is simple: it acts as a middleman. Your ephemeral batch job pushes its metrics to the Pushgateway via HTTP just before exiting. The Pushgateway caches those metrics in memory. Prometheus then continuously scrapes the Pushgateway at its own pace.

Problem solved, right? Not quite. This is where most batch observability architectures begin to rot. To understand why, we have to talk about dimensions.

Understanding Dimensions and Cardinality

In Prometheus, data isn’t just stored as a flat list of numbers. It’s stored as Time Series , defined by a metric name and a set of key-value pairs called labels (or dimensions).

For example, a task failure metric might look like this: airflow_task_status{dag_id="daily_etl", task_id="load_users", status="failed"}

Every unique combination of labels creates a brand new time series in the database. The total number of unique time series is called cardinality.

Low-Cardinality Labels: dag_id (maybe 50 total), task_id (maybe 200 total), status (success, failed, running). These are bounded. They are safe.
High-Cardinality Labels: run_id (a unique UUID for every single execution), user_id, or error_message. These are unbounded. They grow infinitely.

When monitoring a batch workload, you naturally want to know: Which DAG? Which task? Which specific run? How many rows did it process? So, engineers intuitively add a run_id label to their Pushgateway metrics.

The Dumb Proxy and The Race Condition

If you don’t use a unique label like run_id, you hit an immediate wall.

The Pushgateway is essentially a dumb proxy. It doesn’t aggregate metrics, it doesn’t add numbers together, and it doesn’t deduplicate. It simply acts as a key-value store. The “key” is the combination of your labels (the grouping key).

If ten parallel Airflow tasks fail at the exact same millisecond, and they all push failure_count = 1 to the Pushgateway without a run_id, they all write to the exact same key. The Pushgateway just overwrites the value ten times. Prometheus scrapes it once and sees: failure_count = 1. You just lost nine failure records.

This is a race condition. Parallel tasks overwrite each other because they share the same grouping key.

The Fatal Fix: Fighting Prometheus

The “obvious” fix is to add run_id to the grouping key. Because every run has a unique ID, every task gets its own isolated slot in the Pushgateway. The race condition is solved!

But you’ve just created a ticking time bomb.

By adding run_id, you have introduced unbounded cardinality. Every single task execution creates a brand new time series. Furthermore, the Pushgateway has no native TTL (Time To Live). It holds every metric group in memory forever.

Within a few days of high-volume DAG runs, the Pushgateway’s memory balloons until the container OOM (Out of Memory) crashes. When Prometheus tries to scrape it, the massive payload crashes the scraper.

This happens because teams misunderstand what Prometheus is. Prometheus is designed for low-cardinality, continuous metrics. It is not designed to store high-cardinality, event-based data. When you push per-run execution data into Prometheus, you are treating a time-series database like an event log.

The Realization: Two Different Problems

The breakthrough happens when you realize that “batch observability” isn’t a single problem. You are actually trying to answer two completely different questions:

Operational Metrics: “Is the system healthy right now? What is the overall failure rate?” (Needs aggregation, low-cardinality).
State Snapshots: “What is the current status of this specific task run?” (Needs per-run isolation, high-cardinality).

Trying to force both of these through the Pushgateway is the core architectural mistake. We need two different data paths.

The Architecture Stack

Here is the robust, Staff-level architecture we use to solve this cleanly:

Prometheus: Time-series database — stores aggregated operational metrics and temporary task state snapshots.
Grafana: Dashboarding and alerting — visualizes metrics from Prometheus.
StatsD Exporter: The missing link — catches high-frequency UDP bursts, aggregates them, and safely exposes them to Prometheus.
Pushgateway: Push-based ingestion — restricted only to temporary state snapshots.
Sweeper DAG: Automated cleanup — deletes stale Pushgateway metric groups to prevent OOM crashes.

Path 1: Operational Metrics via StatsD

Airflow Task → UDP packet → StatsD Exporter → Prometheus scrape → Grafana

StatsD solves the race condition at the network layer. It listens over UDP. When 10 tasks fire airflow.task.failed:1|c at the exact same millisecond, StatsD catches all 10 packets in memory, adds them up, and flushes a single aggregated metric (failure_count = 10) to Prometheus.

No unique identifiers (run_id) are needed. No race conditions. Zero cardinality explosion. Best of all, Airflow has StatsD support built into its core—you just have to enable it.

Path 2: Task State Snapshots via Pushgateway

Airflow Plugin → HTTP POST → Pushgateway → Prometheus scrape → Grafana

StatsD is perfect for counts, but it can’t answer: “What is the current state of task X in run Y?”

For per-task state visibility at a small-to-medium scale, we do use the Pushgateway, and we do use run_id as a grouping key. However, we apply strict lifecycle management. We deploy an Airflow Sweeper DAG that runs on a schedule, queries the Pushgateway REST API, and deletes metric groups older than a configured threshold.

This prevents OOM crashes while giving us exactly enough runway to monitor active DAGs.

The Architectural Boundary: It is critical to understand that this Pushgateway approach is a stepping stone. At a truly massive scale (tens of thousands of tasks per hour), Prometheus ingestion will still choke on the cardinality, and the Sweeper DAG itself becomes a bottleneck. For large-scale environments, you must abandon Prometheus for per-run tracking entirely and push task events to a dedicated event log system (like Elasticsearch, Grafana Loki, or ClickHouse).

Key Takeaways

Prometheus is not an event log: It is built for low-cardinality, continuous metrics. Do not use it for durable, per-run audit trails.
The Pushgateway is a dumb proxy: Pushing concurrent metrics without unique keys causes race conditions. Adding unique keys causes OOM crashes.
StatsD is the missing link: It aggregates concurrent events at the network layer, completely eliminating race conditions and cardinality bloat for operational metrics.
Scale dictates your state architecture: Use the Pushgateway + Sweeper DAG for medium-scale task state snapshots. For massive scale, move per-run state tracking entirely to an event logging system like Elasticsearch or Loki.

References

The Green Tick Fallacy — Why Batch Observability is Fundamentally Different

Chirag Bhatia — Sun, 17 May 2026 06:40:21 +0000

The Green Tick Fallacy

There is a dangerous assumption that every junior data engineer makes: If the Airflow task turns green, the job was successful.

This is the “Green Tick Fallacy.” When your Spark job finishes, Airflow checks exactly one thing: did the container return an exit 0 status code? It has absolutely no idea if your job processed 10 billion rows flawlessly, or if it processed 0 rows because an upstream partition was empty. It just knows the container didn’t crash.

Relying on the green tick is how you get paged at 3 AM for silent data corruption. To build true batch workload observability, you have to extract internal application metrics — and doing that for batch workloads is fundamentally harder than for services.

Why Batch Observability is Hard

Traditional microservice observability is straightforward. The service runs 24/7, exposing a /metrics HTTP endpoint. Prometheus scrapes it every 15 seconds. The process is always alive to respond.

Batch jobs are ephemeral. They spin up, chew through a terabyte of data in 45 seconds, and vanish. By the time Prometheus tries to scrape them, the process is already dead. You cannot scrape batch jobs — you must push telemetry from inside the code out to an aggregator before the container dies.

This creates a fundamentally different architectural challenge. In the world of services, your observability tool pulls data. In the world of batch, your job pushes data. And the tools designed for pulling don’t work cleanly for pushing.

What Observability Actually Means for Batch

Before diving into tools, it’s worth defining what “observability” actually means for batch workloads. There are three distinct categories of data you need, and each demands a different architectural approach:

1. Operational Metrics

“Is the system healthy right now?”

How many DAGs ran today?
What’s the average task duration?
How many tasks failed in the last hour?

These are low-cardinality, aggregate numbers. You don’t need per-run-id granularity. You need rates, counts, and histograms. These are the bread and butter of Prometheus.

2. Task State Snapshots

“What is the current state of this specific task?”

Is task load_customers in the daily_etl DAG currently running, failed, or succeeded?
What was the duration of this specific execution?
Did the task retry, and what is its final state?

These are point-in-time state snapshots with moderate-to-high cardinality. Each task execution has a unique identity (run_id), and the state may change over the lifecycle (running → failed → retried → success). These can live in Prometheus temporarily, but require careful lifecycle management.

3. Execution History & Audit

“What exactly happened in run XYZ?”

How many rows did run_id=abc123 process?
What was the data quality score for this specific schema version?
What was the exact error message and stack trace?

This is high-cardinality, per-execution, durable data. It must be queryable weeks or months later for debugging, reconciliation, and compliance. This data does not belong in Prometheus.

The Trap: Forcing Everything into One Tool

The mistake most teams make is trying to force all three categories into a single observability system — usually Prometheus via the Pushgateway. This leads to:

Pushgateway abuse : Pushing per-run_id metrics to Prometheus via Pushgateway, creating unbounded cardinality
OOM crashes : Pushgateway has no native TTL, so dynamically labelled metrics accumulate in memory forever
Semantic mismatches : Using the wrong metric type (Counters where Gauges belong), leading to inflated or incorrect dashboard numbers when tasks retry

In this series, we’ll build a clean architecture that uses the right tool for each category. We’ll show you exactly how to set up the stack, what code to write, and what traps to avoid.

What’s Coming in This Series

Part 2: The Architecture — How to design a Prometheus + Grafana + StatsD architecture for batch workloads. What belongs in Prometheus, what doesn’t, and where StatsD fits.
Part 3: Metric Granularity & Classification — What level of observability to achieve where. Why Gauges are semantically correct for batch state (and why it’s NOT about performance). What data belongs in Prometheus vs. structured logs and OLAP stores for auditing and traceability.
Part 4: The Implementation — Building a production-ready Airflow plugin with Gauges, configuring StatsD, designing a Sweeper DAG, and setting up Grafana dashboards. Complete code walkthrough.
Part 5: Future Scope — How to build durable, per-execution audit trails using event streams, OLAP stores, distributed tracing, and data lineage for the data that should never touch Prometheus.

Key Takeaways

For Juniors: The green tick lies. exit 0 means the container didn’t crash, not that it processed data correctly. You must push application metrics out of your batch jobs.
For Seniors: Batch observability has three distinct data categories (operational metrics, state snapshots, execution history). Forcing all three into Prometheus is a common and expensive mistake.
The Rule: Match your data to the right storage backend. Not everything belongs in a time-series database.