DEV Community: sectestai

Why testing broken access control is still mostly a manual job — and what it takes to automate it

sectestai — Wed, 04 Nov 2026 09:00:30 +0000

Broken object-level authorization has held the top spot on the OWASP API Security Top 10 for years now. It's also the class that scanners are worst at, for reasons that have less to do with tooling quality than with what the vulnerability actually is.

I want to lay out why, concretely, because "automate the login and replay the requests" sounds like a weekend project and turns out to be a state-machine problem.

The bug isn't in the response — it's in the meaning of the response
A scanner can tell you that an endpoint returned 200 and a JSON body. It cannot tell you whether that body should have been returned to this caller.

GET /api/invoices/1234 returning a well-formed invoice document looks identical whether or not 1234 belongs to the authenticated user. There's no signature for "you shouldn't be able to read this." You only know it's a bug if you also know:

who the caller is,
who owns object 1234,
and what the authorization model intends.
The first two require real accounts with real data. The third requires a model of the application that isn't in the traffic.

That's the whole difficulty in one sentence: authorization bugs are correct-looking responses.

You need at least two identities that actually exist
Testing this means: act as user A, discover an object ID; act as user B, request the same object; compare.

So the harness needs two accounts. Not two tokens for the same user — two users, each with their own session, their own data, and enough state that the comparison means anything:

A has created an invoice, an order, a comment, whatever the endpoint returns;
B is a separate tenant/team/role, and can reach the same endpoint;
ideally a third identity at a lower privilege level, for function-level checks.
Standing that up per environment is where most of the effort goes. Email verification (an inbox you control), invite flows, org membership, role assignment, seeded fixtures — every application does this differently and most of it is behind UI, not an API.

The object ID usually only exists mid-flow
Real multitenant apps don't put sequential IDs in a list endpoint anymore. The identifier you need to test with shows up somewhere else: in the response to a create call, inside a nested object, in a redirect URL, in a WebSocket frame after a websocket handshake, in a 201 body.

The tester has to carry that value from where it appeared to a different request, in a different session, possibly after a state transition. That's a data-flow problem inside a live application, and it's the part static request lists can't express.

Sessions are a moving target
Between "A creates the object" and "B requests it," the state can change underneath you:

CSRF tokens rotate and are bound to the session cookie;
some apps invalidate other sessions when roles change;
MFA gates certain routes, so the session that can reach the endpoint isn't the session that can log in;
short-lived JWTs expire mid-flow and the client silently refreshes — the harness must too, or it starts sending 401s and reporting them as "no issue found."
Each of these is a real bug surface and a source of false negatives. A harness that quietly loses its session produces a clean report, which is the worst possible outcome.

Deciding whether you found something is a judgment call
Suppose B successfully reads A's invoice. Was that wrong?

Not necessarily. Maybe the endpoint is intentionally shared-scope within an organization. Maybe A and B are in the same tenant and the fixture was wrong — your test data said they were separate, the app disagreed.

Resolving that needs semantics: comparing the two responses and reasoning about what's in them, reading the error bodies, noticing that B got a 200 with an empty array (not a leak) versus a 200 with A's data (a leak). That's the part where a language model genuinely helps — and the part where it can confidently invent a finding that isn't there.

Which forces a design decision: every finding has to be backed by raw request and response evidence, with the reasoning attached, and anything the tool couldn't confirm has to be reported as unconfirmed rather than promoted to "vulnerability." A false positive costs you the trust of the engineers who have to read the report; a false negative costs the customer a breach. Both are expensive, and "AI found 47 issues" is not a deliverable.

You also can't break the thing you're testing
Authorization tests are read-heavy but not read-only. Testing a replay or a state transition means causing it: an order gets approved twice, a refund fires twice, an invite is sent to a real address.

Any automation in this space has to be able to distinguish "I need to see if this is possible" from "I'll just do it and see," and favor strategies that prove the outcome without leaving damage — a second request that's rejected, a state read before and after, a canary object created for the purpose.

What actually made it tractable
The approach that finally worked for us: drive a real browser rather than replay requests. Clicks and forms produce the same session state the application gives a human, including the tokens and redirects that a hand-built request list has to guess at. Then the tester holds two of those sessions and moves objects between them deliberately.

That's the shift — from "send requests and pattern-match responses" to "hold two identities and reason about what one should be able to see." It also happens to be the thing scanners can't do, which tells you where the value is.

I work on SecTestAi, which does exactly this against Web and API targets you own or are explicitly authorized to test — two-session differential testing, evidence attached to every finding, and an explicit "could not confirm" bucket. If you've built similar harnesses, I'd like to hear where yours fell over; the session-expiry false negative was the one that bit us hardest.

Gating an AI agent that tests other people's web apps

sectestai — Wed, 04 Nov 2026 08:53:36 +0000

Most agentic tooling discussions are about capability: what the agent can do, how many steps it can chain, how well it recovers from failure. I want to write about the opposite question, because it's the one that decides whether this class of tool is deployable at all:

How does the system decide it's allowed to act at all?

An AI security-testing agent is an unusual case. It holds session cookies for someone's application, it changes state, it probes access control boundaries, and it does all of that unattended. The interesting failure mode isn't "the agent missed a bug." It's "the agent was pointed at a target nobody actually had permission to test."

Here's what we ended up building around that problem, and why each piece is a mechanism rather than a line in a terms-of-service page.

The declaration comes first, per engagement Before any assessment runs, the operator has to declare the target and their relationship to it. The allowed relations are deliberately narrow:

owned — it's your own application
entrusted — a client or employer asked you to test it
There's no "other" option and no free-text justification field that can absorb a lie. If your situation doesn't fit one of those two, the product has no path for you, which is the point.

The declaration isn't a modal you dismiss once. It belongs to a specific ticket, for a specific set of targets, and it's what the rest of the pipeline hangs off.

A different person confirms it The declaration is created by the operator. It has to be confirmed by a human in the portal before the agent will run against the targets.

This split matters more than it looks. An autonomous system that can author its own authorization is an autonomous system with no authorization. The confirmation is a separate step with its own record — who confirmed, when, from where — and the agent has no code path that can produce it.

Location is checked at run time, and a ticket can be withdrawn Scope isn't only "which hostnames." It's also "where is this being done from," and that can change after approval. Target resolution is checked when the run actually starts rather than when the ticket was created, and a confirmed ticket can be revoked — after which the same run stops being authorized.

The takeaway for anyone building similar systems: authorization has a lifetime. If you evaluate it once at the beginning of a long-running job, you've authorized whatever happens for the rest of that job.

The audit trail can be re-verified, not just read Every authorization event is written to a hash-chained log: each entry links to the previous one, and the chain can be recomputed and checked later.

"Re-verified" is doing real work in that sentence. A log you can only read tells you what the system claims happened. A chain you can recompute tells you whether the record has been altered since it was written — by us, or by anyone with database access.

There's a way to complain If an assessment shows up against something you didn't authorize, there has to be a channel that reaches a human, with a stated response time. A product in this category without one is asking to be trusted on the strength of its marketing copy.

What the agent actually does
For context, since the controls only make sense if you know what's being controlled: the agent drives a real browser — clicks, forms, sessions — so it follows user journeys rather than replaying a list of static requests. It reasons about roles, states and business intent, which is where most of the interesting exposure lives: broken object-level authorization, fields returned without masking, approval steps that can be skipped or reordered, state changes that can be raced.

It's an assessment and risk-evaluation tool, not a source-code audit. It also reports what it could not confirm, which I've come to think of as the more honest half of the output.

One account, one operator Each account operates its own assessments and its own targets. There's no shared pool of targets, no "test whatever other customers are testing," no resale of someone else's scope. Boring, but it's the thing that makes the audit trail mean anything.

The general lesson
If you're shipping anything agentic that touches a system you don't own — a browser agent, a scraper, a data-migration assistant — the same three questions apply:

Who decided this was allowed, and is that decision separable from the thing acting?
Does that decision have an expiry, and can it be withdrawn mid-flight?
Can someone independent verify what happened afterwards?
Capability is the easy part. The systems that survive contact with legal, procurement and incident response are the ones with answers to those three.

We're building this at SecTestAi — an authorized Web & API assessment agent with the controls above applied to every run. It's early: prepaid usage, no subscription, and we'd rather hear what's missing from the model than post a case study we can't stand behind yet. If you're building something in the same space, I'd genuinely like to compare notes in the comments.