Keeping (large) data providers organized in PHPUnit

When using data providers for you PHPUnit tests, it's easy to get carried away and add loads of different test cases with subtle differences in parameters. Let's not have a discussion on whether or not your subject under test is doing too many things if you need that many test cases, let's instead focus on how we can keep those test cases readable, understandable and manageable.

Assume say you are writing a test for the following piece of code, it sets three flags on a presentation model class depending on the current state of a session (should a primary action be shown, should a login button be shown and should an upgrade button be shown):

/**
 * @param ServerRequestInterface $request
 * @param PresentationModel      $presentationModel
 *
 * @return PresentationModel
 */
public function present(ServerRequestInterface $request, PresentationModel $presentationModel): PresentationModel
{
    // Set defaults
    $showLogin = true;
    $showUpgrade = false;
    $showPrimaryButton = true;

    $session = $this->sessionProvider->getSession();
    // Get users loggedIn status and default to false, in case the session is fresh
    $isLoggedIn = (bool)$session->get('loggedIn', false);

    if ($isLoggedIn) {
        $showLogin = false;
        $showUpgrade = true;
    }

    // Check if the user is logged in *and* a pro user
    if ($isLoggedIn && $session->get('subscriptionLevel', Subscription::LEVEL_BASIC) === Subscription::LEVEL_PRO) {
        $showUpgrade = false;
        $showPrimaryButton = false;
    }

    return $presentationModel->withVariables([
        'show_login' => $showLogin,
        'show_upgrade' => $showUpgrade,
        'show_primary_button' => $showLogin || $showPrimaryButton,
    ]);
}

We'll write three test cases for now:

• It's a first visit to the website and the session is still empty
• It's a visit where the logged in status is set to false and there is no subscriptionLevel set in the session
• It's a visit where the logged in status is set to true and the subscriptionLevel status is PRO.

We'll be writing the data provider first:

/**
 * @return array[]
 */
public function sessionDataProvider(): array
{
    return [
        [
            [],
            true,
            false,
            true,
        ],
        [
            ['loggedIn' => false],
            true,
            false,
            true,
        ],
        [
            ['loggedIn' => true, 'subscriptionLevel' => Subscription::LEVEL_PRO],
            false,
            false,
            false,
        ],
    ];
}

Would you care to guess what those undescriptive booleans mean? You'll probably be able to figure that out, but it'll definitely take longer than necessary. How about we make it a bit more descripive? Remember that PHPUnit will simply take the values of the arrays returned by the data provider in the order in which they are defined, it doesn't care much about array keys.

/**
 * @return array[]
 */
public function sessionDataProvider(): array
{
    return [
        [
            'session' => [],
            'showLogin' => true,
            'showUpgrade' => false,
            'showPrimaryButton' => true,
        ],
        [
            'session' => ['loggedIn' => false],
            'showLogin' => true,
            'showUpgrade' => false,
            'showPrimaryButton' => true,
        ],
        [
            'session' => ['loggedIn' => true, 'subscriptionLevel' => Subscription::LEVEL_PRO],
            'showLogin' => false,
            'showUpgrade' => false,
            'showPrimaryButton' => false,
        ],
    ];
}

This way it's immediately clear what those values represent. When you're returning to this test, six months from now, you won't have to find the test implementation first to find the meaning of [true, false, true].

There is still room for improvement though. Even though it's clear what the variables mean, it's not immediately clear what we're testing. We could do better and one way of doing it would be to provide the data set with an array key too:

/**
 * @return array[]
 */
public function sessionDataProvider(): array
{
    return [
        'fresh-session' => [
            'session' => [],
            'showLogin' => true,
            'showUpgrade' => false,
            'showPrimaryButton' => true,
        ],
        'not-logged-in-subscription-unknown' => [
            'session' => ['loggedIn' => false],
            'showLogin' => true,
            'showUpgrade' => false,
            'showPrimaryButton' => true,
        ],
        'logged-in-pro-user' => [
            'session' => ['loggedIn' => true, 'subscriptionLevel' => Subscription::LEVEL_PRO],
            'showLogin' => false,
            'showUpgrade' => false,
            'showPrimaryButton' => false,
        ],
    ];
}

This again helps in the readability of your data provider. You won't ever have to think "why did I/someone add this test case, what is it even testing?".

The data provider is now pretty readable, let's quickly implement the test itself:

/**
 * @dataProvider sessionDataProvider
 *
 * @param array  $sessionData
 * @param bool   $showLogin
 * @param bool   $showUpgrade
 * @param bool   $showPrimaryButton
 *
 * @return void
 */
public function testHeaderShouldBeShownWithCorrectButtonAction(
    array $sessionData,
    bool $showLogin,
    bool $showUpgrade,
    bool $showPrimaryButton
): void {
    $this->sessionProvider->getSession()->willReturn(Session::fromData($sessionData));

    $presentationModel = $this->presenter->present($this->request, $this->presentationModel);

    $this->assertEquals($showLogin, $presentationModel->getVariable('show_login'));
    $this->assertEquals($showUpgrade, $presentationModel->getVariable('show_upgrade'));
    $this->assertEquals($showPrimaryButton, $presentationModel->getVariable('show_primary_button'));
}

Very straightforward and you'll see the name of the data set is now also available in the test output, immediately indicating which test case failed:

There were 3 failures:

1) TestClass::testHeaderShouldBeShownWithCorrectButtonAction with data set "fresh-session" (array(), true, false, true)
Failed asserting that false matches expected true.

/path/to/TestClass.php:88

2) TestClass::testHeaderShouldBeShownWithCorrectButtonAction with data set "not-logged-in-subscription-unknown" (array(false), true, false, true)
Failed asserting that false matches expected true.

/path/to/TestClass.php:88

3) TestClass::testHeaderShouldBeShownWithCorrectButtonAction with data set "logged-in-pro-user" (array(true, 'pro'), false, false, true)
Failed asserting that false matches expected true.

/path/to/TestClass.php:88

We're getting extremely close now, but we can step it up one more notch, because Failed asserting that false matches expected true is still not as helpful in debugging as it could be, we have multiple assertions that could expect true but are now getting false. Let's improve our setup again: instead of using a description of the test case as an array key in the data provider, let's make it part of the returned data set, because fortunately PHPUnit allows passing a custom error message to assertions.

/**
 * @return array[]
 */
public function sessionDataProvider(): array
{
    return [
        [
            'testCase' => 'Fresh Session',
            'session' => [],
            'showLogin' => true,
            'showUpgrade' => false,
            'showPrimaryButton' => true,
        ],
        [
            'testCase' => 'Not Logged In & Subscription Unknown',
            'session' => ['loggedIn' => false],
            'showLogin' => true,
            'showUpgrade' => false,
            'showPrimaryButton' => true,
        ],
        [
            'testCase' => 'Logged In & PRO subscription',
            'session' => ['loggedIn' => true, 'subscriptionLevel' => Subscription::LEVEL_PRO],
            'showLogin' => false,
            'showUpgrade' => false,
            'showPrimaryButton' => false,
        ],
    ];
}

/**
 * @dataProvider sessionDataProvider
 *
 * @param string $testCase
 * @param array  $sessionData
 * @param bool   $showLogin
 * @param bool   $showUpgrade
 * @param bool   $showPrimaryButton
 *
 * @return void
 */
public function testHeaderShouldBeShownWithCorrectButtonAction(
    string $testCase,
    array $sessionData,
    bool $showLogin,
    bool $showUpgrade,
    bool $showPrimaryButton
): void {
    $this->sessionProvider->getSession()->willReturn(Session::fromData($sessionData));

    $presentationModel = $this->presenter->present($this->request, $this->presentationModel);

    $this->assertEquals(
        $showLogin,
        $presentationModel->getVariable('show_login'),
        "`show_login` set incorrectly for test case \"{$testCase}\""
    );
    $this->assertEquals(
        $showUpgrade,
        $presentationModel->getVariable('show_upgrade'),
        "`show_upgrade` set incorrectly for test case \"{$testCase}\""
    );
    $this->assertEquals(
        $showPrimaryButton,
        $presentationModel->getVariable('show_primary_button'), 
        "`show_primary_button` set incorrectly for test case \"{$testCase}\""
    );
}

This would be the output if the test fails now:

There were 3 failures:

1) TestClass::testHeaderShouldBeShownWithCorrectButtonAction with data set #0 ('Empty Session', array(), true, false, true)
`show_login` set incorrectly for test case "Empty Session"
Failed asserting that false matches expected true.

/path/to/TestClass.php:88

2) TestClass::testHeaderShouldBeShownWithCorrectButtonAction with data set #1 ('Not Logged In', array(false), true, false, true)
`show_login` set incorrectly for test case "Not Logged In"
Failed asserting that false matches expected true.

/path/to/TestClass.php:88

3) TestClass::testHeaderShouldBeShownWithCorrectButtonAction with data set #2 ('Not Logged In & Basic Subscription', array(false, 'basic'), true, false, true)
`show_login` set incorrectly for test case "Not Logged In & Basic Subscription"
Failed asserting that false matches expected true.

/path/to/TestClass.php:88

This makes it very clear at first glance:

• which test failed
• which test case for that test failed
• which assertion in that particular scenario failed

Conclusion

Describe your test cases and test parameters in order to be able to still maintain/read/debug your test somewhere in the future. Also make sure you help your future self by providing descriptive messages when your tests/assertions fail. You'll thank your past self when the time comes.

Comments

[Byron Nagi]

Fun fact an alternate way to set the test case name -- instead of using setName in the test -- is to put the name of the test as the key value in the data provider. For example:

public function sessionDataProvider(): array
{
    return [
        'Fresh Session' => [
            'session' => [],
            'showLogin' => true,
            'showUpgrade' => false,
            'showPrimaryButton' => true,
        ]
    ];
}

Failing tests should output 'Fresh Session' instead of 'data set #0'

[Erik Booij]

You're absolutely right, I didn't realize that before and also didn't notice it when writing up the code samples 🤦‍♂️ I've updated the post to reflect this. Thanks!

[Tomáš Votruba]

Data providers are even better organized with yield instead of return array. It also gives you the ability to re-use array in items.

I switched with Rector in all my tests and I cannot go back :)

[Erik Booij]

Hi Tomáš, I must admit that I haven't given generators in data providers much thought yet, so it's at least a welcome new perspective.

As of yet, I don't see any real clear advantages of generators over plain old arrays though. Syntax is a bit different and one might argue that a yield looks a bit cleaner, whereas one might also argue that more developers are likely to be familiar with arrays than generators, which might help collaboration. Am I missing some benefits?

Thanks for the Rector hint. I hadn't heard of it before and will give it a chance for sure 👌