Postgres Row-Level Security doesn't raise an error when it blocks you. It returns zero rows. Your query succeeds. Your agent sees an empty queue. Your agent idles. Your jobs pile up.
This is what happened to ARIA, our autonomous AI system at Elevare Digital.
Key Takeaways
- Postgres RLS silently filters rows on
SELECT— a blocked read and a genuinely empty table look identical to the caller - An orchestrator that reads its own queue gets no exception, no warning, no non-200 status code when RLS blocks it
- A healthy heartbeat on an idle agent tells you nothing about whether the idle is real
- The fix: add a canary count that verifies you can read, not just that you did read
- Treat an empty read as a question, not an answer
What the agent saw
The orchestrator polls a work queue table. Normal operation:
- Query for pending jobs
- If results → process them
- If empty → log idle heartbeat, sleep
The logs showed healthy idle heartbeats. Monitoring showed the agent alive and polling. From the outside, everything looked fine.
Jobs were not being processed.
What actually happened
Someone added a Row-Level Security policy to the queue table, scoped to auth.uid(). Correct for user-facing reads. But the orchestrator connects under the service role — and no service-role bypass was added to the policy.
Here's the policy as it was written:
-- Added for user-facing queue visibility
CREATE POLICY "users_see_own_jobs"
ON work_queue
FOR SELECT
USING (auth.uid() = user_id);
And RLS was enabled on the table:
ALTER TABLE work_queue ENABLE ROW LEVEL SECURITY;
No bypass for the service role. No FOR ALL escape. The orchestrator's SELECT now matched zero rows — because RLS filtered every row out before returning results.
Postgres does not raise. It does not warn. The query completes with status 200 and an empty array.
// Orchestrator poll — looks completely normal
const { data: jobs, error } = await supabase
.from('work_queue')
.select('*')
.eq('status', 'pending');
// error is null. jobs is []. Agent concludes: nothing to do.
if (!jobs || jobs.length === 0) {
await recordIdleHeartbeat();
return;
}
The error is null. The jobs array is empty. The agent's logic is correct for the data it received. The data it received was wrong in a way that produced no signal.
Why this is hard to catch
If RLS blocks a write, you often get an error — the row you tried to insert violates a policy, or returns nothing when you expected a rowcount. Writes are easier to notice.
SELECT under RLS is different. The design intent is that users should not be able to distinguish "this row doesn't exist" from "this row exists but you can't see it." That's a security feature. It's also exactly the wrong behavior for an orchestrator that needs to know the difference between "queue is empty" and "I am blind."
The service role in Supabase bypasses RLS by default — but only if you're using the service-role key on the client. If your edge function or backend is using the anon key, or if it's operating in a context where auth.uid() resolves to null, RLS applies and silently filters.
-- This is what the orchestrator needed, but wasn't there
CREATE POLICY "service_role_bypass"
ON work_queue
FOR ALL
USING (auth.role() = 'service_role');
-- Or, simpler: grant the service role explicit bypass at the table level
ALTER TABLE work_queue FORCE ROW LEVEL SECURITY; -- applies to all
-- and then in your Supabase client: use the service-role key, which bypasses RLS automatically
The actual bypass mechanism in Supabase is that the service-role JWT includes "role": "service_role", and Supabase's Postgres config grants that role BYPASSRLS. If your client is initialized with the service-role key, you're fine. If it's not, RLS applies — no error, just silence.
The fix: a canary count
The real repair has two parts: fix the RLS policy, and add a check that will catch this failure mode again.
Part 1 — Fix the policy:
-- Preserve user-facing policy
CREATE POLICY "users_see_own_jobs"
ON work_queue
FOR SELECT
USING (auth.uid() = user_id);
-- Add explicit service-role access
-- (Or: initialize your orchestrator client with the service-role key,
-- which bypasses RLS automatically via BYPASSRLS grant)
CREATE POLICY "orchestrator_full_access"
ON work_queue
FOR ALL
USING (auth.role() = 'service_role')
WITH CHECK (auth.role() = 'service_role');
Part 2 — The canary check:
The orchestrator now runs a canary query before trusting an empty result. The canary queries a row it knows exists — a sentinel row inserted specifically for this purpose, or a count from an unrestricted table the orchestrator owns.
async function pollQueue() {
const { data: jobs, error: queueError } = await supabase
.from('work_queue')
.select('*')
.eq('status', 'pending');
if (queueError) {
await alertOpsChannel('queue_poll_error', queueError);
return;
}
// Jobs came back — process normally
if (jobs && jobs.length > 0) {
await processJobs(jobs);
return;
}
// Empty result — but is it genuinely empty, or are we blind?
const canaryOk = await verifyReadAccess();
if (!canaryOk) {
// This is the case we were missing before
await alertOpsChannel('queue_read_access_lost', {
message: 'Orchestrator received empty queue result but canary check failed. Possible RLS or permission regression.',
});
return; // Do NOT record idle heartbeat — we don't know the real state
}
// Canary passed — empty really means empty
await recordIdleHeartbeat();
}
async function verifyReadAccess(): Promise<boolean> {
// Query a sentinel row that always exists in a known state
// This could be a dedicated canary table, or a system-health row
const { data, error } = await supabase
.from('orchestrator_canary')
.select('id')
.eq('id', 'heartbeat-sentinel')
.single();
if (error || !data) {
// We expected exactly one row. Getting nothing means access is broken.
return false;
}
return true;
}
The canary table is simple:
CREATE TABLE orchestrator_canary (
id TEXT PRIMARY KEY
);
INSERT INTO orchestrator_canary (id) VALUES ('heartbeat-sentinel');
-- No RLS on this table — it exists only so the orchestrator can verify it can read
-- If you want RLS: add only a service-role policy, nothing user-facing
Now the orchestrator has three states instead of two:
- Jobs found → process
- No jobs, canary ok → genuinely idle
- No jobs, canary failed → alert, do not idle
The third state was always real. We just had no way to observe it.
The deeper issue with autonomous agents and silent failures
Human-in-the-loop systems fail loudly. A user tries to load their queue, sees nothing, and files a ticket. An autonomous agent has no user. It reads, decides, acts. If the read is silently wrong, the decision is wrong, and nothing complains.
This failure mode matters more as systems get more autonomous. ARIA processes queued work without someone watching every poll cycle. The assumption baked into the polling loop was: an empty read means there is nothing to do. That assumption held until it didn't, and the system had no way to question it.
The fix is to make the orchestrator skeptical of its own empty results. Not paranoid — just skeptical enough to verify the precondition that makes "empty" meaningful.
What to check in your own system
If you're running any kind of queue worker against Postgres or Supabase:
Check which key your worker is using. Supabase service-role key bypasses RLS. Anon key does not. Confirm which one is in your edge function or backend environment.
List your RLS policies and check for missing bypasses.
SELECT * FROM pg_policies WHERE tablename = 'your_queue_table';Look at your idle heartbeats. If idle logging increased around the time you last modified permissions or added RLS, that's worth investigating.
Add a canary. Takes an hour. Catches this entire class of problem permanently.
An empty queue result is not evidence of an empty queue. It's evidence that the query ran. Those are different things, and in autonomous systems, the difference matters.
— Mike Clarke, founder of Elevare Digital.
Top comments (0)