DEV Community

Cover image for In an audit, a green test means the bug is real
Jules Robineau
Jules Robineau

Posted on Originally published at jrobineau.com

In an audit, a green test means the bug is real

An audit report says "this route lets anyone find out which accounts exist". The team reads it, nods and moves on to the next ticket. A test shows the request and the response. Anyone can run it again on their own machine.

TL;DR: in an audit, I write tests that pass as long as the flaw exists. A green test proves the problem, with the exact request and response. Everything runs on an isolated test database, with made-up accounts, never touching production. Once the flaw is fixed, the test is flipped and becomes a regression test. The report keeps its paragraphs for what the tests do not prove.

This article is for developers and leads who order a security audit, or receive one. No security background needed.

The setup: a white-box audit of an API

In September, I ran a white-box audit of an API. Two questions drove the work. Can someone without an account find out who is registered? Can a logged-in user touch another account's data?

The core of the audit was five HTTP tests. An HTTP test sends a real request to the API and checks the response. The five held 21 assertions. An assertion is one precise check, like "the status code is 200".

All five tests were green. That was the bad news.

Here I cover the public routes, the ones that work without an account. The other tests targeted another account's IDs sent in the request body, the topic of a separate article.

An audit test passes when the flaw exists

Usually a green test means the code works. I have written before that a green test only proves what it actually checks. In an audit, I make it check the dangerous behavior on purpose.

Each test describes the flaw as if it were the expected behavior. For example, "with a wrong token, the API returns the profile". If the test passes, the flaw is real. If it fails, the flaw was not reproduced under these conditions. Check the fixtures, configuration and actual response before drawing a conclusion.

That is the big difference with a paragraph. A paragraph can be wrong and nobody notices. A test makes the evidence reproducible. Tests can be wrong too, so review their assertions.

The example: "forgot password" answers 200 or 422

Take the "forgot password" route. You send an email address, and the API sends a link to change the password. Nothing could be more ordinary.

In this API, the response depended on the address. A registered address got a 200 status code, which means "OK". An unknown address got a 422, which means "request rejected".

That difference is enough for account enumeration. Someone tries a list of addresses and keeps the ones that answer 200. They now know who has an account, without any password.

And the route had no explicit rate limit. Nothing slowed down mass attempts.

Here is the audit test, simplified for this article. The API was written in PHP with Laravel, but the idea works in any language.

// Audit test: green means the flaw exists.
public function test_forgot_password_reveals_which_emails_exist(): void
{
    User::factory()->create(['email' => 'known@example.test']);

    $this->postJson('/auth/forgot-password', ['email' => 'known@example.test'])
        ->assertStatus(200);

    $this->postJson('/auth/forgot-password', ['email' => 'unknown@example.test'])
        ->assertStatus(422);
}
Enter fullscreen mode Exit fullscreen mode

The test creates a fake account, then sends two requests. It passes if the two status codes differ. Green means the flaw exists.

This is not a rare case. In 2026, several CVEs describe exactly the same flaw. Here are two, checked in the official registry.

  • CVE-2026-24468, in the open source project OpenAEV. The /api/reset route answered 400 for an unknown email and 200 for a valid one. It is fixed in version 2.0.13.
  • CVE-2026-58503, in the Frappe web framework. The enumeration went through the reset_password route. It is fixed in versions 15.106.0 and 16.16.0.

Everything happens far from production

An audit must not cause the incident it is looking for. So no test touched production. Not a single request went to it.

The tests ran in a temporary Docker environment. That environment had no access to the outside network. Even a mistake in a test could not reach a real server.

The database was an isolated MySQL test database. The accounts and records were synthetic. Synthetic means made up for the test. No real person is behind them.

During the audit, I did not change the project's code. The proof tests were not added to its repository.

This setup has one more benefit. Anyone on the team can replay the tests, with no production access and no real data.

The report says what the tests do not prove

A test proves one precise thing, and nothing else. The report has to say so. That is where paragraphs really earn their place.

Take the "verify email" route. For an active account, it returned the profile with any non-empty token. That profile also held the account's internal ID.

Combined with "forgot password", this made a chain. You test addresses, then you collect the IDs of the accounts you found. Yet the report states the limit. It works one address at a time, from a guessed list. The audit found no route that hands out every account at once.

The report also splits two lists. On one side, flaws confirmed by a test. On the other, leads seen in the code but not tested. A lead stays a lead until a test confirms it.

Finally, the tests cover the local code, not the deployed application. And the project already had a test checking that one account was denied access to another's data. It passed too. Its green said nothing about the routes my tests targeted.

Once the flaw is fixed, the test flips

The cycle has four steps: reproduce the flaw, fix the code, see the proof fail, then write the protective test.

The audit test passes as long as the flaw exists, then flips

Green proves the flaw, then the same test keeps it closed. During the audit, the test passes as long as the route answers 200 or 422 depending on the address. After the fix, it expects the same answer for both.

A green audit test is an alarm. It should not stay green. After the fix, it fails, and that is the signal you want.

You then rewrite it the other way around. It becomes a regression test that replays the same request on every code change. For "forgot password", both addresses must receive the expected generic response. Two identical errors are not enough.

// Regression test: green means the flaw stays closed.
public function test_forgot_password_answers_the_same_for_every_email(): void
{
    \Illuminate\Support\Facades\Notification::fake();
    $user = User::factory()->create(['email' => 'known@example.test']);

    $known = $this->postJson('/auth/forgot-password', ['email' => 'known@example.test']);
    $unknown = $this->postJson('/auth/forgot-password', ['email' => 'unknown@example.test']);

    // This example's API contract is 200 with this exact generic response.
    $expected = ['message' => 'If the account exists, a reset link will be sent.'];
    $known->assertStatus(200)->assertExactJson($expected);
    $unknown->assertStatus(200)->assertExactJson($expected);
    \Illuminate\Support\Facades\Notification::assertSentTo(
        $user, \Illuminate\Auth\Notifications\ResetPassword::class
    );
    \Illuminate\Support\Facades\Notification::assertCount(1);
}
Enter fullscreen mode Exit fullscreen mode

This example uses Laravel’s standard notification, intercepted by Notification::fake(). Adapt the status, message and notification to your API. This test does not measure response time. OWASP also asks for a constant response time. Otherwise the delay gives the address away. The rate limit deserves its own test too.

I did this flip on the "verify email" route. After the fix, I added a test that sends a wrong token. It expects a 422 rejection, with no profile data in the response. A second test checks that an unknown account gets the same error.

Why a test beats a paragraph

A report gets read once. A test lives in CI and runs again on every change. It keeps working after the auditor has left.

The test establishes the observed behavior. Severity still depends on the exposed data, required permissions and deployed configuration.

Finally, it sets the expected result without dictating the fix. The team chooses how to fix it. The test only says when it is done.

The checklist

In an audit, every confirmed flaw gets its test. Once the flaw is fixed, that test flips and stays in CI.

  • [ ] Write each suspected flaw as a test that passes if the flaw exists
  • [ ] Run the tests on an isolated test database, never on production
  • [ ] Use only made-up accounts and data
  • [ ] Cut network access from the test environment
  • [ ] Keep flaws confirmed by a test apart from leads seen in the code
  • [ ] Write in the report what each test does not prove
  • [ ] On "forgot password", compare the status code and message for a known and an unknown email
  • [ ] Test response time and rate limiting separately
  • [ ] After the fix, flip each test and keep it in CI

The takeaway

In an audit, green changes meaning. It no longer says "it works". It says "the flaw is here, and here is how to replay it".

Then the same test switches sides. Flipped, it keeps the flaw closed, long after the audit.

Want an audit that delivers proof your team can replay? See my audit and hardening offer, or let's talk.


Sources: Laravel, testing notifications · CVE-2026-24468 (OpenAEV) · OpenAEV, advisory GHSA-v6rg-hf9w-f8ph · CVE-2026-58503 (Frappe) · OWASP, Forgot Password Cheat Sheet · OWASP WSTG, testing for account enumeration · CWE-204, Observable Response Discrepancy

Top comments (0)