DEV Community

Mike Clarke
Mike Clarke

Posted on

The function was deployed, healthy, and never ran once

#webdev #programming #learning #coding

There's a category of bug where nothing is broken. The dashboard is green. The deployment succeeded. The function shows a recent update timestamp. And it has done zero work since you shipped it.

We hit this in ARIA, our autonomous agent system. A scheduled function was silently doing nothing — not erroring, not timing out, just... absent.

Key Takeaways

  • A Supabase edge function can be perfectly deployed and still never execute
  • JWT verification happens at the gateway, before your handler runs
  • Gateway-level 401s produce no application logs, no error rows, no telemetry inside the function
  • The only signal is the absence of invocation-log entries
  • Functions called by pg_cron need verify_jwt=false if they authenticate via service-role key in the body

What we saw

ARIA runs a multi-agent scheduler. Several edge functions are invoked on a schedule via pg_cron. One of them showed up in the Supabase dashboard as deployed, healthy, recently updated. We had no alerts firing.

What we didn't have: any invocation log rows. Not failed ones. Not slow ones. Zero rows.

Everything looked fine. Nothing was running.

The mechanism

Supabase edge functions have a verify_jwt flag. When it's set to true (the default), the gateway validates the bearer token on every request before the function body executes.

Our pg_cron job was calling the function with a service-role bearer token — correct credentials, correct format. But the gateway's JWT verification was rejecting it and returning a 401 before the handler ever ran.

The request never reached the function. The function had nothing to log.

Here's what the pg_cron invocation looks like:

select cron.schedule(
  'run-agent-task',
  '* * * * *',
  $$
  select net.http_post(
    url := 'https://<project>.supabase.co/functions/v1/agent-task',
    headers := jsonb_build_object(
      'Content-Type', 'application/json',
      'Authorization', 'Bearer ' || current_setting('app.service_role_key')
    ),
    body := '{}'::jsonb
  );
  $$
);
Enter fullscreen mode Exit fullscreen mode

And here's the function config that caused the silent failure:

# supabase/functions/agent-task/config.toml
[functions.agent-task]
verify_jwt = true  # <-- gateway rejects the cron call here, handler never runs
Enter fullscreen mode Exit fullscreen mode

The fix is one line:

[functions.agent-task]
verify_jwt = false  # gateway passes the request through; function handles auth itself
Enter fullscreen mode Exit fullscreen mode

With verify_jwt=false, you move the authentication check inside the function body, where you can actually see what happens:

import { serve } from 'https://deno.land/std@0.168.0/http/server.ts'

serve(async (req) => {
  const authHeader = req.headers.get('Authorization')
  const token = authHeader?.replace('Bearer ', '')

  // Validate against your service role key inside the handler
  // where failures are visible in your logs
  if (token !== Deno.env.get('SUPABASE_SERVICE_ROLE_KEY')) {
    console.error('Unauthorized request to agent-task')
    return new Response('Unauthorized', { status: 401 })
  }

  // actual work happens here
  console.log('agent-task executing')
  // ...

  return new Response('ok', { status: 200 })
})
Enter fullscreen mode Exit fullscreen mode

Now when auth fails, you get a log line. You get an invocation row. You get something to look at.

Why this is hard to catch

The normal debugging loop assumes that if something failed, there's a record of the failure. You look at logs, you find the error, you fix it.

Gateway-level rejections break that loop. The 401 happens in infrastructure you don't control and don't have direct log access to. From the function's perspective, the request never arrived. From the cron job's perspective, it fired successfully — net.http_post queued the request and moved on.

The only diagnostic signal is the shape of what's missing: no invocation rows in the function's log table, for a function that should be running every minute.

If you're not actively checking for expected invocations, you won't notice.

What we added after

Beyond fixing the flag, we added a simple liveness check: a separate monitor queries the invocation log and alerts if a scheduled function hasn't produced a row in longer than two of its scheduled intervals. Not sophisticated, but it catches the "deployed and doing nothing" class of failure.

-- Example: check that agent-task has run in the last 3 minutes
-- (for a function scheduled every minute)
select
  case
    when max(created_at) < now() - interval '3 minutes'
    then 'STALLED'
    else 'OK'
  end as status
from function_invocation_log
where function_name = 'agent-task';
Enter fullscreen mode Exit fullscreen mode

The specific table structure depends on how you surface invocation logs in your setup — the point is to query for expected evidence of execution, not just absence of errors.

The thing worth internalizing

Deployment success and runtime success are different things. A function can pass every health check and still never execute a single line of your code.

For any function invoked by infrastructure (cron, queues, webhooks) rather than by a user, you need positive confirmation of execution — not just the absence of errors. Absence of errors and absence of execution look identical from the outside.

Check your cron-invoked functions. If verify_jwt=true and you're calling with a service-role token, check the invocation log. The rows might not be there.

— Mike Clarke, founder of Elevare Digital.

Top comments (0)