DEV Community

Cover image for Characterization Tests Before You Refactor AI-Written Code
Muhammad Ali Mughal
Muhammad Ali Mughal

Posted on

Characterization Tests Before You Refactor AI-Written Code

#laravel #php #testing #ai

A client asked us to "clean up" an invoicing module that had been written mostly with an AI assistant over a few frantic weeks. It worked. Customers were billed correctly. But the main service class was 1,400 lines, had three nearly identical rounding helpers, and nobody on the team could say which one was actually used for which tax rule. The tempting move was to ask an assistant to refactor it. We did not, not yet. First we spent two days writing tests that recorded exactly what the code did today, including the parts that looked wrong. On day four of the refactor, one of those tests caught a rounding change that would have shifted about 1 in 300 invoices by a cent. Finance would have found it at month end.

I wrote this with help from AI tools and checked every snippet myself against Laravel 11, PHPUnit 11 and Pest 3. I am a Principal Software Engineer in Karachi with 9+ years across PHP/Laravel, Vue, Node and .NET microservices, and through Loop Origin I maintain production SaaS codebases where a growing share of code is AI-assisted.

📚 Series Navigation — Maintaining AI-Written Code in Production

  • Part 1: Reviewing AI-Generated Pull Requests: A Checklist That Catches Real Bugs
  • Part 2 (This Post): Characterization Tests Before You Refactor AI-Written Code
  • Part 3: Hallucinated Packages & Dependency Drift in AI-Assisted Code
  • Part 4: Static Analysis Gates: PHPStan, TypeScript Strict & Coverage in CI
  • Part 5: Decision Records So AI Context Doesn't Rot
  • Part 6: Measuring AI Coding Impact With DORA Metrics, Not Vibes

Related earlier posts: Cron Jobs That Never Ran: Heartbeats & Dead-Man Switches · Choosing Redis, SQS or Database Queues for Laravel and Node SaaS

What is a characterization test and why write one before refactoring AI-written code?

Direct answer: A characterization test records what code currently does, not what it should do. Before refactoring AI-written code, capture its real outputs for representative inputs (golden files, snapshots or exact assertions), freeze time, randomness and IDs, run the tests until they are stable, then refactor in small steps and treat any test change as a behaviour change to review.

Rows of labelled glass test tubes in a metal rack under cool light, AI-generated

Why AI-written code needs this more, not less

Michael Feathers named characterization tests in Working Effectively with Legacy Code: tests that describe actual behaviour so you can change code safely. AI-assisted code is a special kind of legacy code. It often arrives fast, looks tidy, and has no author who remembers the reasoning. Three patterns show up again and again in the codebases we maintain:

Pattern Why it is risky to refactor What the characterization test pins
Duplicate helpers with tiny differences You merge them and quietly change edge cases Output of each caller for boundary inputs
Implicit ordering (queries, collections) A cleaner query returns rows in a different order Exact order of exported or displayed rows
Silent fallbacks (?? 0, empty catch blocks) Removing them changes what users see on bad data Behaviour with nulls, missing relations, bad input
Time- and locale-dependent logic Tests pass today, fail at month end or in another zone Output with frozen time and fixed locale
Over-broad side effects Refactor drops an event, email or cache write Dispatched jobs, events, mails and DB writes

The goal is not to bless the current behaviour as correct. It is to make every change visible, so the team decides on purpose.

Step 1: Choose the seams and the inputs

Do not try to test the whole module. Pick the public entry points that the rest of the app calls, usually a service method, a controller action or a job's handle(). For the invoicing module that was InvoiceCalculator::calculate() and the GenerateMonthlyInvoices job.

For inputs, I mine production shapes, never production data. We wrote a query that counted combinations of tax region, discount type, currency and line-item count, then built anonymized fixtures that covered the top combinations plus known edge cases: zero-quantity lines, 100% discounts, three-decimal currencies, and a customer with no billing address.

Step 2: Make the code deterministic

Characterization tests that flicker are worse than none. Freeze everything that changes between runs:

use Illuminate\Support\Carbon;
use Illuminate\Support\Str;

beforeEach(function () {
    Carbon::setTestNow('2026-10-31 23:59:00');      // month-end is where bugs hide
    Str::createUuidsUsingSequence([
        '00000000-0000-0000-0000-000000000001',
        '00000000-0000-0000-0000-000000000002',
    ]);
    app()->setLocale('en');
    config(['app.timezone' => 'UTC']);
    mt_srand(42);
});

afterEach(function () {
    Carbon::setTestNow();
    Str::createUuidsNormally();
});
Enter fullscreen mode Exit fullscreen mode

If the code calls rand(), now() or uniqid() deep inside, that is a finding in itself. Note it, and wrap it behind something you can control during the refactor.

Step 3: Capture golden files

For outputs with many fields, assertions written by hand miss things. I serialize the full result to JSON and compare it against a committed golden file:

function assertMatchesGolden(string $name, mixed $actual): void
{
    $path = base_path("tests/golden/{$name}.json");
    $json = json_encode($actual, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES | JSON_PRESERVE_ZERO_FRACTION);

    if (getenv('UPDATE_GOLDEN') === '1' || ! file_exists($path)) {
        file_put_contents($path, $json . "\n");
        test()->markTestIncomplete("Golden file written: {$name}");
    }

    expect($json . "\n")->toBe(file_get_contents($path));
}

it('characterizes invoice totals per fixture', function (string $fixture) {
    $customer = loadFixture($fixture);            // builds models from anonymized JSON
    $invoice = app(InvoiceCalculator::class)->calculate($customer, period: '2026-10');

    assertMatchesGolden("invoice/{$fixture}", [
        'lines'    => $invoice->lines->map->only(['sku', 'qty', 'unit', 'tax', 'total'])->all(),
        'subtotal' => $invoice->subtotal,
        'tax'      => $invoice->tax,
        'total'    => $invoice->total,
        'currency' => $invoice->currency,
    ]);
})->with(fn () => array_map('basename', glob(base_path('tests/fixtures/invoice/*.json'))));
Enter fullscreen mode Exit fullscreen mode

Two rules make golden files useful. First, keep them readable and committed, so a diff in a PR shows exactly which number changed. Second, never regenerate them in the same commit as a refactor without a reviewer looking at the golden diff. UPDATE_GOLDEN=1 is a decision, not a shortcut.

Step 4: Pin side effects, not just return values

The invoicing job also sent emails, dispatched a PDF job and wrote an audit row. A refactor that drops one of those still returns the right totals. Laravel's fakes make side effects easy to pin:

it('characterizes side effects of monthly invoicing', function () {
    Mail::fake();
    Queue::fake();
    Event::fake([InvoiceIssued::class]);

    $customers = seedFixtureCustomers(['eu_vat', 'us_no_tax', 'zero_qty_line']);
    (new GenerateMonthlyInvoices('2026-10'))->handle();

    Mail::assertSent(InvoiceMail::class, 3);
    Queue::assertPushed(RenderInvoicePdf::class, 3);
    Event::assertDispatchedTimes(InvoiceIssued::class, 3);
    expect(AuditLog::where('action', 'invoice.issued')->count())->toBe(3);
});
Enter fullscreen mode Exit fullscreen mode

Step 5: Check that the tests can actually fail

Coverage tells you the code ran, not that your tests would notice a change. Before refactoring, I run mutation testing on the target class. Infection mutates the code (flips operators, changes rounding, removes calls) and reports which mutants survived:

vendor/bin/infection --filter=app/Billing/InvoiceCalculator.php \
  --threads=4 --min-msi=80 --show-mutations
Enter fullscreen mode Exit fullscreen mode

Surviving mutants point directly at behaviour your characterization suite does not pin. In our case, a surviving mutant in one rounding helper is how we found that two of the three helpers were only reached by the zero-quantity path, which we then added fixtures for.

Step 6: Refactor in small, reviewable steps

With the suite green and stable, the refactor becomes boring, which is the goal:

  1. One mechanical change per commit (extract method, rename, merge duplicates).
  2. Run the characterization suite on every commit.
  3. If a golden file changes, stop and decide: bug fix (update golden, write it in the PR description) or regression (revert).
  4. Keep behaviour changes in separate PRs from mechanical ones, as Part 1 of this series recommends.

This is the same discipline we use on Laravel maintenance and refactoring work, whether the original author was a person or an assistant.

Diagnostic checklist

  • [ ] Public entry points (services, actions, jobs) for the module are identified.
  • [ ] Fixtures cover the top real input shapes plus known edge cases, with no production data.
  • [ ] Time, UUIDs, randomness, locale and timezone are frozen in tests.
  • [ ] Golden files are committed, readable and reviewed when they change.
  • [ ] Side effects (mail, queue, events, audit rows) are asserted.
  • [ ] The suite passes 10 times in a row with no flakes.
  • [ ] Mutation score on the target class is checked and surviving mutants are reviewed.
  • [ ] Refactor commits are small and mechanical, with behaviour changes split out.

FAQ

Are characterization tests the same as snapshot tests?

Snapshot tests are one way to write them. The idea is broader: any test that records current behaviour, whether through golden files, snapshots or exact assertions. What matters is that changes are visible and reviewed.

What if the current behaviour is clearly a bug?

Pin it anyway, and mark it with a comment or a test name like it('currently rounds half down (bug #412)'). Fix it in a separate PR so the behaviour change is explicit.

Can I ask an AI assistant to write the characterization tests?

Yes, it is good at scaffolding fixtures and boilerplate. Review the inputs it chooses, because it tends to pick happy paths. You still decide which edge cases matter.

How many fixtures are enough?

Enough that mutation testing kills most mutants in the code you plan to touch. For the invoicing module that was 38 fixtures. A small helper might need five.

Should these tests stay after the refactor?

Keep the valuable ones and convert them into normal behaviour tests with clear names. Delete golden files that only pinned internal details you no longer care about.

Is this worth it for a small module?

If a wrong result costs money or trust, yes. For a small module it is often an afternoon of work, which is much cheaper than a month-end incident. If you want help planning it, get in touch.

Next in this series, Part 3 covers hallucinated packages and dependency drift in AI-assisted code.

Follow & Connect

Top comments (0)