The release looked clean. Every feature test green, coverage reported at 90%, sign-off given on a Thursday. On Monday a support ticket arrived: a user on a viewer role had opened a record belonging to a different team, through a URL a colleague pasted into chat. The button that would have taken them there was hidden. The endpoint behind it was not.
Nothing in that suite was wrong. The login test passed because the user could log in. The record-detail test passed because an admin could open a record. No test ever asked the only question that mattered — what happens when this identity calls that endpoint — because no one had written the question down.
This is the shape of almost every permission defect we see on B2B SaaS products with roles, tenants, and plan tiers. It is not a coverage problem. It is a state-explosion problem.
Coverage is counted over features; permissions live in the product of role × resource × action × tenant configuration, and nobody puts that product in the denominator. A feature can be genuinely 100% covered under one role and completely untested under the other nine.
OWASP's 2025 Top 10 keeps Broken Access Control at #1, and the underlying data is blunt: 100% of the applications in the dataset showed some form of broken access control, across 1,839,701 occurrences and 32,654 mapped CVEs. The API edition rates broken object-level authorization "Widespread" in prevalence and "Easy" in detectability — which is the useful part. These bugs are cheap to find if you look, and they ship anyway because the suite is organized by feature and the bug lives in identity.
Why permission bugs survive a suite you trust
A test suite is a map of the product's features; an authorization model is a map of its identities. Passing tests on the first says close to nothing about the second.
Three failure modes account for most of what reaches production. The first is hidden UI treated as security: the frontend removes a button based on a role claim, and nobody asserts that the server refuses the same call. The second is role change mid-session — a user demoted from editor to viewer keeps a token minted under the old grants, and every test that starts from a fresh login misses it. The third is permission inherited from tenant configuration rather than from the user record: the role is member in both tenants, but one tenant has an integration connected or a plan tier that quietly widens what member may do.
Scale rules out brute force. DeviQA's engagement on an art management and sales platform is a fair illustration: the product ships permission controls alongside cataloguing, advanced search, order and sales management, exhibitions and presentation tooling, and the whole thing has been supported since 2023 by a single manual QA engineer who built 515 test cases, reached 90% test coverage and logged over 1,000 bugs. At that ratio you do not test a role matrix by clicking it. You either enumerate and prune it deliberately, or you sample it by accident.
One caveat worth stating plainly: no public source breaks that project's defects down by category, so treat the mechanism above as reasoning about how suites are structured, not as a measured finding about which bugs escaped.
The unit of testing is not the workflow
Stop thinking in workflows and start thinking in cells. The unit is the tuple (role, resource, action, tenant setting), and it belongs in a file the whole team can read:
resources:
invoice:
actions: [list, read, create, update, void, export]
roles:
owner: [list, read, create, update, void, export]
admin: [list, read, create, update, void]
member: [list, read, create]
viewer: [list, read]
guest: []
modifiers:
billing_module_disabled: { member: [], viewer: [] }
plan_tier_free: { owner: [list, read, create, update] }
Six actions × five roles is thirty cells for one resource, before modifiers. Twenty resources is six hundred. You do not test all of them, and the pruning rules matter more than the enumeration: collapse roles with provably identical grant sets into one representative and assert the identity of the sets once; keep every boundary between adjacent roles, because that is where product decisions are argued and where they drift; test inheritance once per level rather than once per screen.
The first time a team writes this file, the interesting output is not the test plan. It is the arguments. Cells nobody can answer are the spec gaps, and they are usually the same cells the defects come from — can a member void an invoice they created? Does viewer see the export button, the export endpoint, or neither?
That structural work is where the returns are. On the art platform above, the fix was process, not tooling: a redesigned Jira workflow, a standardized user story template, mandatory reproduction context — 100% of bug reports now carry screenshots or video — and a cleanup of stale test cases. Lost or unclear tickets fell 95%, early-stage defect detection rose 60%, ticket reopen rate halved, and defect reproduction time dropped 30%. Those figures cover the whole QA process, not permission testing specifically, and the team never published a permission matrix — but the ambiguity that hides permission cells is the ambiguity those changes removed.
Negative tests belong at the API layer
Every cell where the answer is "no" is one cheap HTTP request. Not a browser session, not a login, not a page object — a request with a token and an assertion on two things: the status code, and the absence of data in the body.
// one row per forbidden cell, generated from the matrix file
for (const { role, action, endpoint } of forbidden(matrix, 'invoice')) {
it(${role} cannot ${action} invoice, async () => {
const res = await api(endpoint, { token: tokenFor(role, TENANT_A) });
expect(res.status).toBe(403); // not 200, and not 500
expect(JSON.stringify(res.body)).not.toMatch(/amount|customer_id|iban/);
});
}
// cross-tenant: a perfectly valid token, pointed at someone else's object
it('owner of tenant B cannot read tenant A invoice', async () => {
const res = await api(/invoices/${TENANT_A_INVOICE}, { token: ownerOf(TENANT_B) });
expect([403, 404]).toContain(res.status);
});
The body assertion is the one people skip and the one that catches real leakage — a 403 that still returns the object in an error payload is a breach with a polite status line. Decide as a team whether cross-tenant reads answer 403 or 404 and assert the choice; an inconsistent answer is itself an enumeration oracle for an attacker.
Test transitions, not just fresh sessions: mint a token as editor, demote the user, replay the same call, assert the new answer. That is the mid-session case that per-role E2E runs structurally cannot see.
The economics are the argument. On Compass, a team of eight full-stack QA engineers plus one manual engineer, working the product since 2016, built 500+ API auto-tests across multiple endpoints alongside 1,600+ automation scripts; automated E2E per pull request went from 40+ minutes to 10, regression from 7 days to 2, and test coverage from 50% to 85%. Hundreds of API assertions fit in the time one UI login takes. Rentgrata shows why the cross-tenant half matters: the product splits into a renter-facing Messenger and a property-manager Manager over shared data — two audiences, one dataset — and since 2020 a two-person QA team has produced 9,000+ test cases (6,000+ web, 3,000+ mobile), 2,500+ automation tests and 90% coverage. Neither team published a permission matrix or negative-access suite; they are evidence about layering and cost, not about authorization method.
The limit: API-level checks assume the API is the only path to the data. Server-rendered views, GraphQL resolvers, exports, webhooks and background jobs each need their own assertions, or you have proved something about one door in a building with five.
What the UI suite is still for
The UI suite keeps a real but small job: asserting conditional rendering — which elements each role actually sees — and one end-to-end pass per privilege tier, not per role. If admin and owner differ by two grants, they are one tier for E2E purposes and two rows in the API matrix.
Generated UI coverage is worth knowing about here, with its ceiling stated. On SuperbCompanies, two manual QA engineers running since 2024 used OwlityAI — autonomous UI testing without access to source code — to reach a reported 92% coverage across UI features and 1,200+ test cases, with vendor-reported outcomes of 70% fewer production bugs and 90% lower QA costs. Treat those percentages as directional: they are engagement-reported figures for a vendor tool, with no published baseline. And the structural point stands regardless of the number — a generator that infers tests from the rendered application has no model of what the application should forbid. It can tell you a screen still renders. It cannot tell you a cell should have been a 403.
Test data is the actual bottleneck
Most teams under-test roles for an unromantic reason: producing a user in role R, tenant T, state S takes four minutes of clicking, so the suite quietly settles for the two identities that already exist.
Fix provisioning before you expand the matrix. Seed through an internal API or directly into the database, generate identities per test rather than sharing fixtures, and clean up after:
const ctx = await seed.tenant({ plan: 'pro', modules: ['billing'] });
const viewer = await seed.user(ctx, { role: 'viewer' });
const invoice = await seed.invoice(ctx, { status: 'open' });
// ...assertions...
afterEach(() => seed.destroy(ctx)); // tenant-scoped teardown, no shared state
Sprinklr is the reference point for how much this buys: over five years a team of seven rebuilt an unreliable suite, added prerequisite test data directly to the database, and ran 16 threads across multiple machines — ending at ~2,000 automated tests, >90% coverage, 10,000+ test cases, ~12,000 bugs reported (45% major) and 50% less testing time. The 50% is not attributable to seeding alone; the framework, the threading and the CI integration changed together. MINT shows the environment side: splitting one large team into ~15 squads each with its own sandbox, plus a dedicated UAT environment, moved releases from one every 1–2 months to 4–5 per day, and feature lead time from 1–2 months to two weeks, across 800+ releases.
The trap in DB seeding is specific: inserting rows straight into the permissions tables bypasses the code that assigns permissions. If invitation, role change, SSO group mapping and plan downgrade are only ever exercised by your seeder, a bug in the assignment path is invisible to every test that depends on it. Keep a small set of tests that provision through the real assignment API and assert the resulting grants.
Tenant settings and integrations are hidden role modifiers
"Admin" is not one identity. It is one identity per combination of feature flags, plan tier, and connected integrations — each of which can add or remove capability without touching the user record. A connected integration usually brings its own data scope and its own callbacks, and a role that cannot see a field in the UI may be able to read it through the integration's sync. Compass, for instance, is tested against third-party integrations including Twilio, with load testing up to 1,000 concurrent users; Planetly runs its 400+ E2E tests on 4 parallel threads across 3 browsers and 2 localizations, keeping automated smoke under 10 minutes and regression under an hour with >90% coverage.
Both are configuration axes multiplying an existing suite — the arithmetic a role matrix already faces. Add the axis explicitly, as a modifiers block, and prune it with the same rules. No public source itemizes defects caused by tenant configuration, so treat this as an axis to enumerate, not a measured category.
What this costs
The matrix is a permanent maintenance obligation. Every new role, plan tier, integration and resource re-opens it, and stale cells are worse than missing ones because they read as verified. Negative tests fail in a particularly nasty way: rename an endpoint and every "asserts 403" test keeps passing against a 404 from a route that no longer exists. Assert that the positive counterpart of each forbidden call returns 200 for the role that should have it, or the suite decays into green nothing.
None of this is fast. The engagements above run from one to ten years with dedicated staffing — Sprinklr five years with seven engineers, Compass since 2016 with nine, MINT since 2019 with 21, Rentgrata since 2020 with two — and none of them reports a number that ties coverage percentage to defect escape or security posture, because none is published.
The honest small-team version is three things: a written matrix that someone owns, negative API assertions on every boundary between adjacent roles, and one UI pass per privilege tier. Cut roles-per-run before you cut negative assertions — a second full E2E pass under a second role buys far less than fifty 403 assertions.
What that plan leaves uncovered, stated plainly: non-adjacent role pairs, most modifier combinations, every path that is not the API, and the assignment code itself unless you test it separately. Those are known gaps rather than surprises, which is the entire difference between a permission bug you decided not to chase and one that arrives as a support ticket on a Monday.
Top comments (0)