Agentic PHPUnit output

#php #ai #testing

I was made a aware of PAO. And while it think it is a good tool I think we can do better by making it more useful for an LLM.

The package has options for PHPUnit, Pest and ParaTest. I'm only going to focus on PHPUnit, version 12 in particular.

The setup

PHPUnit has an option to add extensions. The best way to let PHPUnit know your extension is in the phpunit.xml file.

<extensions>
    <bootstrap class="Tests\Extensions\AgentAwareOutputExtension"/>
</extensions>

To detect when PHPunit is run inside an agent I used the shipfastlabs/agent-detector library (I saw it in PAO). This library uses well known config variables to detect multiple agents. Because I'm trying out Mistral Vibe now I added a new script to composer.json.

"test:agent": "AI_AGENT=1 vendor/bin/phpunit"

While PAO uses json as output, I want to use markdown.
From the documentation I got that it doesn't show the errors. Which strikes me as odd because you want your coding agent to be able to fix the failing tests, not? So that is on my todo list.

The code

In the PHPUnit I saw an example where the used a intermediate class to collect the needed data, so that is what I did.

class TestDataCollector
{
    public function __construct(
        public int $failed = 0,
        public int $passed = 0,
        public int $total = 0,
        public array $messages = [],
    )
    {}

    public function write() : void
    {
        $text = '# Test results'. PHP_EOL . PHP_EOL;
        $text .= '## Summary' . PHP_EOL. PHP_EOL;
        $text .= 'failed: ' . $this->failed . PHP_EOL;
        $text .= 'passed: ' . $this->passed . PHP_EOL;
        $text .= 'total: ' . $this->total . PHP_EOL;

        if(count($this->messages) > 0) {
            $text .=  PHP_EOL . '## Failed tests' . PHP_EOL. PHP_EOL;
            $text .= '| Test | Message |' . PHP_EOL;
            $text .= '| --- | --- |' . PHP_EOL;

            foreach($this->messages as $message) {
                $text .= '| ' . $message['test'].' | ' . $message['message'] . ' |' . PHP_EOL;
            }
        }


        fwrite(STDOUT, $text);
    }
}

In the constructor I setup all the properties I needed to for PHPUnit to manipulate them based on the status of the tests.

The write method is used to display the result of the tests.
I choose to use concatenation to make it easy to maintain.

Next up is the extension, the glue that holds the different parts together.

use AgentDetector\AgentDetector;
use PHPUnit\Runner\Extension\Extension;
use PHPUnit\Runner\Extension\Facade;
use PHPUnit\Runner\Extension\ParameterCollection;
use PHPUnit\TextUI\Configuration\Configuration;
use Tests\Extensions\Subscribers\FailSubsrciber;
use Tests\Extensions\Subscribers\ErrorSubscriber;
use Tests\Extensions\Subscribers\TestsDoneSubscriber;
use Tests\Extensions\Subscribers\PassSubsrciber;

class AgentAwareOutputExtension implements Extension
{
    public function bootstrap(Configuration $configuration, Facade $facade, ParameterCollection $parameters): void
    {
        if ($configuration->noOutput()) {
            return;
        }

        $agentDetector = new AgentDetector();

        if (!$agentDetector->detect()->isAgent) {
            return;
        }

        $facade->replaceOutput();
        $facade->replaceProgressOutput();
        $facade->replaceResultOutput();

        $testDataCollector = new TestDataCollector();

        $facade->registerSubscribers(
            new PassSubsrciber($testDataCollector),
            new FailSubscriber($testDataCollector),
            new ErrorSubscriber($testDataCollector),
            new TestsDoneSubscriber($testDataCollector),
        );
    }
}

A better way would be to have the extension in its own directory, but for the purpose of the test I kept the directory structure flatter.

The Extension has a single method bootstrap, which mirrors the configuration in phpunit.xml.

PHPUnit has a --no-output CLI option that is why the first lines in bootstrap exist.

The AgentDetector lines are, as you can guess, to create a guard when an AI agent is not detected.

The replace methods are a bit unfortunately named because they prevent the display of the default output.

PHPUnit has quite a few Subscriber interfaces for all the events that can happen. So it is up to us to pick the ones we need.

use PHPUnit\Event\Test\Passed;
use PHPUnit\Event\Test\PassedSubscriber;
use Tests\Extensions\TestDataCollector;

class PassSubsrciber implements PassedSubscriber
{

    public function __construct(private TestDataCollector $testDataCollector)
    {}

    public function notify(Passed $event): void
    {
        $this->testDataCollector->passed++;
        $this->testDataCollector->total++;
    }
}

Because I don't need much data you will see most subscribers have little content.

use PHPUnit\Event\Test\Failed;
use PHPUnit\Event\Test\FailedSubscriber;
use Tests\Extensions\TestDataCollector;

class FailSubscriber implements FailedSubscriber
{

    public function __construct(private TestDataCollector $testDataCollector)
    {}

    public function notify(Failed $event): void
    {
        $this->testDataCollector->failed++;
        $this->testDataCollector->total++;

        $this->testDataCollector->messages[] = [
          'test' => $event->test()->className().'::'.$event->test()->methodName(),
          'message' => $event->throwable()->message(),
        ];
    }
}

The main difference between this subscriber and the next one is that this catches the failed tests and the next one the PHP errors.

use PHPUnit\Event\Test\Errored;
use PHPUnit\Event\Test\ErroredSubscriber;
use Tests\Extensions\TestDataCollector;

class ErrorSubscriber implements ErroredSubscriber
{

    public function __construct(private TestDataCollector $testDataCollector)
    {}

    public function notify(Errored $event): void
    {
        $this->testDataCollector->failed++;
        $this->testDataCollector->total++;

        $this->testDataCollector->messages[] = [
            'test' => $event->test()->className().'::'.$event->test()->methodName(),
            'message' => $event->throwable()->message(),
        ];
    }
}

The last subscriber is where the output happens.

use PHPUnit\Event\TestRunner\Finished;
use PHPUnit\Event\TestRunner\FinishedSubscriber;
use Tests\Extensions\TestDataCollector;

class TestsDoneSubscriber implements FinishedSubscriber
{

    public function __construct(private TestDataCollector $testDataCollector)
    {}

    public function notify(Finished $event): void
    {
        $this->testDataCollector->write();
    }
}

And this can now give an example output of

# Test results

## Summary

failed: 1
passed: 24
total: 25

## Messages

| Test | Message |
| --- | --- |
| App\Tests\AnswerTest::testFail | Failed asserting that false is true. |