Your cron logged success every day for weeks. It approved nothing.

#webdev #programming #learning #coding

We run ARIA, an autonomous content pipeline at Elevare Digital. Agents generate drafts, a daily approver cron reviews them, approved content moves to publishing. Fully automated, gated by a human-review layer, self-healing in most places.

For several weeks, the approver cron ran on schedule, logged { status: 'success' }, and approved zero drafts. Nineteen drafts sat in the queue. Nobody was alerted. The system looked healthy.

Key Takeaways

A cron that processes zero rows and a cron that processes zero rows because there's nothing to process produce identical logs.
PostgREST inner joins silently exclude rows when the join condition matches nothing — no error, no warning, just an empty result set.
Producer/consumer type drift is invisible until you instrument the gap between "rows waiting" and "rows processed."
Alert on pending > 0 AND processed === 0, not just on thrown errors.
Autonomous systems fail most dangerously in the space between errors and correctness.

What the system was doing

ARIA's generator agents write content and store drafts as rows in a content_opportunities table. Each row has an opportunity_type — in practice, the generator was producing 'article' and 'listing' types.

The approver cron queries that table via Supabase / PostgREST and processes any pending drafts it finds. Here's the shape of what it was doing:

// approver-cron/index.ts (Deno edge function)

const { data: opportunities, error } = await supabase
  .from('content_opportunities')
  .select(`
    id,
    title,
    status,
    content_threads!inner(
      id,
      body
    )
  `)
  .eq('status', 'pending_review')
  .eq('opportunity_type', 'thread'); // <-- the problem

if (error) {
  console.error('Approver query failed', error);
  return new Response('error', { status: 500 });
}

console.log(`Approver run complete. Processed: ${opportunities.length}`);
// ... process opportunities

Two things combined to create the silent failure:

opportunity_type = 'thread' — the generator never wrote rows with this type. It wrote 'article' and 'listing'.
content_threads!inner(...) — PostgREST inner join syntax. If no matching rows exist in content_threads, the parent rows are excluded entirely.

The result: opportunities was always an empty array. No error was thrown. opportunities.length logged as 0. The function returned 200 OK.

Why this is the worst kind of failure

Most failures announce themselves. A network timeout throws. A missing env var crashes on startup. A malformed query returns a PostgREST error object.

This failure looked like rest.

In a queue system, there's a legitimate state where the consumer runs and finds nothing to do — because the queue is empty. That's healthy-idle. It logs Processed: 0 and exits cleanly.

Our broken state also logged Processed: 0 and exited cleanly.

From the outside, from a log aggregator, from a dashboard: identical.

// Healthy idle — queue is empty
Approver run complete. Processed: 0

// Broken — 19 drafts waiting, filter excludes all of them  
Approver run complete. Processed: 0

The only way to tell them apart is to check whether the queue actually had pending rows when the consumer ran.

The fix: broaden scope, instrument the gap

The immediate fix was to remove the incorrect type filter and fix the join. The approver should consume all pending drafts, not just ones of a type the generator never produces:

// Fixed: consume what the producer actually writes
const { data: opportunities, error } = await supabase
  .from('content_opportunities')
  .select(`
    id,
    title,
    status,
    opportunity_type
  `)
  .eq('status', 'pending_review');
  // No type filter. No inner join restricting to non-existent rows.

But the more important fix was the alert. After each consumer run, we now check whether there are pending rows that weren't touched:

// After the approver processes its batch:
const processed = opportunities?.length ?? 0;

const { count: stillPending } = await supabase
  .from('content_opportunities')
  .select('id', { count: 'exact', head: true })
  .eq('status', 'pending_review');

if (processed === 0 && (stillPending ?? 0) > 0) {
  // Consumer ran, touched nothing, but the queue has rows.
  // This is the danger zone.
  await triggerAlert({
    type: 'consumer_drift',
    message: `Approver processed 0 rows but queue has ${stillPending} pending drafts.`,
    severity: 'high',
  });
}

console.log(`Processed: ${processed}. Still pending: ${stillPending ?? 0}`);

This check doesn't care why the consumer touched zero rows. It cares that the queue had work and the consumer didn't do it. The reason might be a bad filter, a broken join, a type mismatch, or something else we haven't thought of yet. The alert fires regardless and forces a human to look.

The underlying pattern: producer/consumer drift

The generator and approver were written at different times. At some point, the generator's output types and the approver's filter drifted apart. Neither system knew about the other's contract. There was no shared schema validation between them, no test that asserted "the approver can actually see what the generator writes."

This is a normal way systems decay. The producer evolves. The consumer doesn't. Or vice versa. In a synchronous API call, this kind of mismatch usually causes a visible failure. In a queue-based async pipeline, it just causes the queue to grow while the consumer reports success.

Autonomous systems make this worse because there's no human in the loop watching the queue depth during normal operation. The whole point is that it runs without supervision. Which means the instrumentation is the supervision.

What we'd do differently

A few things that would have caught this earlier:

Log scanned vs. processed separately. If your consumer query returns zero rows, log that distinctly from "query returned rows and we processed them." scanned: 0, processed: 0 vs scanned: 5, processed: 5 are different states.

Track queue depth over time. If pending rows are accumulating across multiple cron runs, that's a signal even without a consumer error. A simple check: if the queue depth at the end of a run is higher than at the start, something isn't working.

Test the consumer against real producer output. Not just "does the function run without throwing" but "given a row the producer actually writes, does the consumer find and process it."

Name your contracts. If the producer writes opportunity_type: 'article' and the consumer filters on opportunity_type: 'thread', that's a broken contract. Treat it like one — define it explicitly, validate it, test it.

The honest summary: a filter we forgot about, combined with an inner join that excluded everything, turned our approver cron into a machine that ran on schedule, logged success, and accomplished nothing for weeks. The fix was straightforward. The lesson is that in autonomous pipelines, success-on-empty is a failure mode you have to design against explicitly, because the logs will never tell you on their own.

Alert on the gap between what the queue holds and what the consumer touches. Not just on errors.

— Mike Clarke, founder of Elevare Digital.