DEV Community

Mike Clarke
Mike Clarke

Posted on

Your pipeline was green for weeks. It shipped nothing.

#programming #coding #webdev #learning

There is a category of production failure that monitoring tools are almost designed to miss. The system runs. The pings arrive. Every dashboard stays green. And somewhere behind that green, nothing is happening.

We hit this with ARIA, the autonomous system we run at Elevare Digital. A pipeline had been silent for weeks. Not crashing — silent. The health checks came in on schedule. The function executed. And the work it existed to do was not getting done.

Key Takeaways

  • Liveness (did it run?) and function (did it accomplish anything?) are different properties. Most health checks only measure one.
  • A cron that reschedules, retries, or re-checks the same item will emit healthy pings indefinitely while producing zero output.
  • The fix is to make your health signal carry a payload: items processed, rows moved, work done.
  • Alert on absence of output over a window, not just on errors.
  • A green heartbeat on a pipeline that produces nothing is a failure wearing a healthy costume.

What we were checking vs. what we needed to check

The health check looked like this conceptually:

// What we had — liveness only
async function runPipeline() {
  await doWork(); // might do nothing
  await recordHealth({ status: 'ok', timestamp: new Date() });
}
Enter fullscreen mode Exit fullscreen mode

The function ran. It called doWork(). It recorded status: ok. Monitoring saw the ping. Everything looked fine.

The problem: doWork() had silently reduced itself to re-checking one item it had already processed. It found nothing new, did nothing, and returned without error. The health signal had no idea. It recorded that the function executed — which was true — and implied from that the pipeline was functioning — which was not.

This distinction sounds obvious written out. It is easy to miss in practice because the failure mode produces no errors, no exceptions, no timeouts. It produces only silence, and your monitoring is not listening for silence.

The mechanism in plain terms

Cron fires. Function wakes up. Function checks for work. No new work is found (or the same old item keeps surfacing and getting skipped). Function exits cleanly. Health ping is recorded.

Repeat. Every interval. For weeks.

From the outside: healthy system.
From the inside: a poster that clocks in, sits down, does nothing, clocks out, and files a timesheet marked "completed."

The fix: make the health signal carry evidence of work

We changed the health record to include output metrics. Not just "did it run" but "what did it produce."

// What we changed to — function-level health
async function runPipeline() {
  const result = await doWork();

  await recordHealth({
    status: 'ok',
    timestamp: new Date(),
    items_processed: result.count, // the actual payload
  });
}
Enter fullscreen mode Exit fullscreen mode

Then we added a check that looks at recent health records and alerts when a supposedly-active pipeline has emitted zero output over a window:

-- Supabase: find active pipelines that reported no work over the last N intervals
SELECT
  pipeline_name,
  COUNT(*) AS runs,
  SUM(items_processed) AS total_output,
  MAX(recorded_at) AS last_run
FROM pipeline_health
WHERE recorded_at > now() - interval '7 days'
GROUP BY pipeline_name
HAVING SUM(items_processed) = 0
ORDER BY last_run DESC;
Enter fullscreen mode Exit fullscreen mode

This query has no interesting results on a healthy system. When it returns rows, a pipeline ran repeatedly and moved nothing — and that is worth an alert regardless of what the status field says.

Why this failure mode is common

Most health check patterns are borrowed from web services, where liveness and function are closely coupled. If your API endpoint returns 200, it almost certainly did the thing it exists to do. The request-response cycle forces the work to happen before the response is emitted.

Background pipelines are different. The work is decoupled from the signal. The function can complete — cleanly, successfully — and still have accomplished nothing. The health check fires after execution regardless of output.

So when teams instrument a pipeline the same way they instrument an endpoint, they end up measuring the wrong thing.

What to add to any pipeline health check

Three fields that matter more than status: ok:

interface PipelineHealthRecord {
  pipeline_name: string;
  recorded_at: string;
  status: 'ok' | 'error';

  // These are the fields that actually tell you something
  items_processed: number; // work done this run
  items_available: number | null; // work seen (optional but useful)
  error_detail: string | null;
}
Enter fullscreen mode Exit fullscreen mode

With items_available you can distinguish between two very different situations:

  • Available = 0, processed = 0: queue is empty, pipeline is idle. Probably fine.
  • Available > 0, processed = 0: work exists, pipeline is not touching it. Not fine.

The second case is what we had. The pipeline could see work (or thought it could), entered its processing loop, and exited without moving anything. Status: ok the whole time.

The alert logic

Once the health records carry real output data, the alert becomes straightforward:

async function checkPipelineOutput(pipelineName: string) {
  const { data } = await supabase
    .from('pipeline_health')
    .select('items_processed, recorded_at')
    .eq('pipeline_name', pipelineName)
    .gte('recorded_at', new Date(Date.now() - 7 * 24 * 60 * 60 * 1000).toISOString())
    .order('recorded_at', { ascending: false });

  if (!data || data.length === 0) return; // no runs at all — separate alert

  const totalOutput = data.reduce((sum, row) => sum + (row.items_processed ?? 0), 0);
  const runCount = data.length;

  if (runCount > 3 && totalOutput === 0) {
    await sendAlert({
      pipeline: pipelineName,
      message: `Ran ${runCount} times in the last 7 days. Produced zero output.`,
      severity: 'high',
    });
  }
}
Enter fullscreen mode Exit fullscreen mode

The threshold (runCount > 3) is tunable. The point is that multiple runs with zero output is the signal. A single zero-output run might be a quiet period. Several in a row is the pipeline telling you something is wrong, if you are listening.

The honest lesson

We were not measuring the wrong thing by accident. We followed a normal health check pattern and it was simply insufficient for this class of job. The pattern works fine for services. It does not work for pipelines.

Log the absence of work as loudly as the presence of errors. Zero output on a pipeline that should be producing is not a quiet success. It is an invisible failure, and the only difference between that and a noisy crash is that the noisy crash gets fixed.

— Mike Clarke, founder of Elevare Digital.

Top comments (0)