My security test was green for six weeks because it could not turn red

I had a SQL test that logged in as two users and checked that neither could read the other's rows. It ran on every push. It was green for about six weeks.

Then I broke an assertion on purpose, just to watch it fail. It didn't fail. It threw a type error on the line that builds the failure message.

Every check in that file had been passing into a branch that had never once executed.

The line

The test collects failures into an array and raises at the end:

DECLARE fails text[] := '{}';
...
IF v_int <> 1 THEN fails := fails || 'own profile not visible'; END IF;
IF v_int <> 0 THEN fails := fails || 'other profiles visible';  END IF;
...
IF array_length(fails, 1) IS NULL THEN
  RAISE EXCEPTION 'PASS';
ELSE
  RAISE EXCEPTION 'FAIL — %', array_to_string(fails, ' | ');
END IF;

fails || 'some message' reads like appending a string to an array, and in Postgres that is a real thing — anyarray || anyelement exists and does exactly that. So the line looks fine, and it type-checks, and it never runs while everything passes.

The moment a check actually fails, it raises:

22P02 malformed array literal: "own profile not visible"

Why

I assumed the message text was the problem — spaces, or something needing quoting. It isn't. Four cases against a real database:

fails := fails || 'own profile not visible';        -- 22P02 malformed array literal
fails := fails || 'own profile not visible'::text;  -- works, {"own profile not visible"}
fails := array_append(fails, 'own profile not visible');  -- works
fails := fails || 'nospaces';                       -- 22P02 malformed array literal

The one with no spaces and no commas fails too, which rules out the content. What actually happens is operator resolution. A bare '...' literal has no type yet — it is unknown. Postgres has both anyarray || anyelement and anyarray || anyarray, and with an unknown on the right it resolves to the array-to-array form, then tries to read your sentence as an array literal.

Add ::text and it picks the other operator. Or skip the ambiguity entirely and use array_append, which is what I did.

None of this is a Postgres bug. It is a perfectly reasonable resolution rule that happens to turn a failure path into a type error, in the one place where nobody looks, because that code only runs on the day something is already wrong.

The part that actually bothers me

The bug is small. What it did was not.

For six weeks I had a green check on a security property, and I would have told you with a straight face that cross-user isolation was covered. It wasn't covered. It wasn't uncovered either — it was unrunnable, and from outside, unrunnable looks exactly like passing. Same colour, same duration, same silence.

The failure mode of a normal bug is that something breaks. The failure mode of this one was confidence.

Two more in the same file, once I started looking

Pulling that thread found two others that had also never fired.

The vacuous pass. One assertion was "user B sees zero of user A's rows." That is trivially true when user A has no rows. Run the suite against a fresh database with one signup and it passes, brilliantly, having tested nothing. It now counts the fixtures first and reports INCONCLUSIVE rather than OK:

SELECT count(*) INTO v_total FROM public.profiles;
IF v_total < 2 THEN
  fails := array_append(fails, format(
    'INCONCLUSIVE: only %s profile(s) exist. Sign up a second user, '
    'otherwise the isolation check passes for lack of data', v_total));
END IF;

That distinction matters more than it sounds. OK and I could not test this are different answers, and a suite that returns the first when it means the second is lying to you politely.

Deny-only. Someone reviewing the file pointed out that every single assertion was of the form "B cannot see A's data." A policy that denies everything to everyone passes all of them. There was no assertion that the thing was still supposed to work, so the suite couldn't tell a correct policy from a broken one that happened to fail closed. That took one line to fix and I have no idea how long I would have gone without noticing.

What I do now

Before trusting a green suite, I break one assertion on purpose and watch it go red. Then I put it back.

It takes ten seconds. It is the only check I have that tells me the difference between a test that passes and a test that cannot fail. Every other signal those two produce is identical.

I would extend it to anything that reports on its own health — a monitor, a linter, a CI gate, a security scanner. If you have never seen it fail, you do not know that it can.