loading...
Cover image for Growing the PHP Core—One Test at a Time

Growing the PHP Core—One Test at a Time

realflowcontrol profile image Florian Engelhardt ・11 min read

In September 2000, I started my vocational training at an internet agency with two teams: one doing JavaServer Pages, and one was doing PHP. I was assigned to the PHP team, and when presented with the language, I immediately knew that no one will ever use this. I was wrong. Today, my entire career is built on PHP. It’s time to give back to the community by writing tests for the PHP core itself!

Changelog

2020-09-23
  • gcov.php.net is retired, and code coverage reports can now be found at Azure Pipelines

Prepare Your machine!

Before you start writing tests for PHP, let’s start with running the tests that already exist. Fetch the PHP source from GitHub and compile it to do so.

$ git clone git@github.com:php/php-src.git
$ cd php-src
$ ./buildconf
$ ./configure --with-zlib
$ make -j `nproc`
Enter fullscreen mode Exit fullscreen mode

I recommend creating a fork upfront because it makes creating a pull request with your test easier, later on.

If you do not have a compiler and build tools already installed on your Linux computer, you should install the development-tools group on Fedora or the build-essential on Debian Linux. The ./configure command may exit with an error condition; this usually occurs when build requirements are not met, so install whatever ./configure is missing and re-run that step. Keep in mind that you need to install the development packages — in my case, the configure script stated it was missing libxml, which was, in fact, installed. What it really missed was the header files, which are in the development package (usually named with a dev or devel suffix). Note that the --with-zlib is mandatory in this case, as you need the zlib extension which is not built by default.

After your build is complete, you can find the PHP binary in ./sapi/cli/php. Go ahead and check what you just created:

$ ./sapi/cli/php –v
PHP 8.0.0-dev (cli) (built: Jul 14 2020 21:05:42) ( NTS )
Copyright (c) The PHP Group
Zend Engine v4.0.0-dev, Copyright (c) Zend Technologies
Enter fullscreen mode Exit fullscreen mode

Now that you have a freshly built and running PHP binary, you can finally run the tests included within the GitHub repository. Since PHP 7.4 tests may run in parallel, you can give the number of parallel jobs with the -j argument.

$ make TEST_PHP_ARGS=-j`nproc` test
...
=============================================================
TEST RESULT SUMMARY
-------------------------------------------------------------
Exts skipped    :   46
Exts tested     :   26
-------------------------------------------------------------
Number of tests : 15670             10668
Tests skipped   : 5002 ( 31.9%) --------
Tests warned    :    0 (  0.0%) (  0.0%)
Tests failed    :    0 (  0.0%) (  0.0%)
Expected fail   :   32 (  0.2%) (  0.3%)
Tests passed    : 10636 ( 67.9%) ( 99.7%)
-------------------------------------------------------------
Time taken      :   76 seconds
=============================================================
Enter fullscreen mode Exit fullscreen mode

This looks good and it only took 76 seconds to run 10636 tests — quite fast. Zero warnings and zero failures, hooray! You can see that 5002 tests have been skipped and 32 have been expected to fail. You will learn about why this is in the next part.

What Do Tests Look Like?

Let’s take a look at what such a test case may look like. PHP test files have the file ending .phpt and consist of several sections, three of which are mandatory: the TEST, the FILE and the EXPECT or EXPECTF section.

--TEST--
strlen() function
--FILE--
<?php
var_dump(strlen('Hello World!'));
?>
--EXPECT--
int(12)
Enter fullscreen mode Exit fullscreen mode

The TEST Section

This is a short description of what you are testing. It should be as short as possible, as this is what is printed when running the tests. You can put additional details in the DESCRIPTION section.

The SKIPIF Section

This section is optional and is parsed by PHP. If the resulting output starts with the word skip, the test case is skipped. If it starts with xfail the test case is run, but it is expected to fail. The primary use case is to check whether a required extension is installed.

The FILE Section

This is the actual test case, PHP code enclosed by PHP tags. This is where you do whatever you want to test. As you have to create output that is then matched against your expectation, you usually find var_dump all over the place.

The EXPECT Section

This section must exactly match the output from the executed code in the FILE section to pass.

The EXPECTF Section

EXPECTF can be used as an alternative to the EXPECT section and allows the usage of substitution tags you may know from the printf functions family:

--EXPECTF--
int(%d)
Enter fullscreen mode Exit fullscreen mode

The XFAIL Section

XFAIL is another optional section. If present, the test case is expected to fail; you don’t need to echo out xfail in the SKIPIF section at all if you have this. It contains a description of why this test case is expected to fail. This feature is mainly used if the test is already finished, but the implementation isn’t yet, or for upstream bugs. It usually contains a link to where a discussion about this topic can be found.

The CLEAN Section

This exists so you can clean up after yourself. An example might be temporary files you created during the test. Keep in mind, this section's code is executed independently from the FILES section, so you have no access to variables declared over there. Also this section is run regardless of the outcome of the test.

You may find more in depth details on the file format at the PHP Quality Assurance Team Web Page.

What Can You Test?

Now that you know what a PHP test looks like it is time to find something to test. Head over to Azure Pipelines, click on the latest scheduled run (look for the calendar icon and the string "Scheduled for") and then click “Code Coverage” to view the code coverage report.

When I started looking for something to test, I found that the zlib_get_coding_type() function in ext/zlib/zlib.c was not covered at all (this was back in the PHP 7.1 branch and on the now retired gcov.php.net).

Code coverage report shows the `zlib_get_coding_type()` function as completely uncovered.

The next step was to check out what this function was supposed to do in the PHP Documentation. What I saw in the documentation, but also in the code itself, was that this function returns the string gzip or deflate or the Boolean value false. The linked zlib.output_compression directive documentation gave one additional bit of information: the zlib output compression feature reacts on the HTTP Accept-Encoding header sent with the client HTTP request.

For the test, this means there are four possible cases to check for:

  • the absence of the Accept-Encoding header
  • the Accept-Encoding being gzip
  • the Accept-Encoding being deflate
  • the Accept-Encoding being anything else

The last case is treated the same as the first case: The function is expected to return the Boolean value false.

Time To Write That Test!

Let's start with the first test in the file test/zlib_get_coding_type.phpt for the case that there is no Accept-Encoding header set.

--TEST--
zlib_get_coding_type()
--SKIPIF--
<?php
if (!extension_loaded("zlib")) print "skip";
?>
--FILE--
<?php
ini_set('zlib.output_compression', 'Off');
var_dump(zlib_get_coding_type());
ini_set('zlib.output_compression', 'On');
var_dump(zlib_get_coding_type());
?>
--EXPECT--
bool(false)
bool(false)
Enter fullscreen mode Exit fullscreen mode

You can run this single test via:

$ make test TESTS=test/zlib_get_coding_type.phpt
Enter fullscreen mode Exit fullscreen mode

Sadly, this gives you a failed test. You can easily see why this is the case, by checking the test directory contents where you find your .phpt test file and some other files with various file endings. One that is particularly interesting, in this case, is the zlib_get_coding_type.log file:

---- EXPECTED OUTPUT
bool(false)
bool(false)
---- ACTUAL OUTPUT
bool(false)
Warning: ini_set(): Cannot change zlib.output_compression - headers already sent in test/zlib_get_coding_type.php on line 4
bool(false)
---- FAILED
Enter fullscreen mode Exit fullscreen mode

And this is your first time learning about PHP internals. You cannot change the output compression setting after your script created any form of output. For this to work, there seems to be a handler that is called when we change the zlib.output_compression directive. Try searching for "zlib.output_compression" in the ext/zlib/zlib.c source code file. You find it in the call to the STD_PHP_INI_BOOLEAN macro along with the pointer to the function OnUpdate_zlib_output_compression in which you can spot the warning you received. To work around this warning, you can change the FILE section:

<?php
ini_set('zlib.output_compression', 'Off');
$off = zlib_get_coding_type();
ini_set('zlib.output_compression', 'On');
$on = zlib_get_coding_type();
var_dump($off);
var_dump($on);
?>
Enter fullscreen mode Exit fullscreen mode

Rerunning the test case results in your first successful test! 🎉

It’s not time to party yet; there are still other cases to cover. The next one on the list is the case with the HTTP Accept-Encoding header set to the string gzip. As a PHP developer, you know you can access and change the HTTP headers via the super global $_SERVER. Let’s change the FILE and EXPECT section accordingly.

--FILE--
<?php
ini_set('zlib.output_compression', 'Off');
$off = zlib_get_coding_type();
ini_set('zlib.output_compression', 'On');
$on = zlib_get_coding_type();
$_SERVER['HTTP_ACCEPT_ENCODING'] = 'gzip';
$gzip = zlib_get_coding_type();
var_dump($off);
var_dump($on);
var_dump($gzip);
?>
--EXPECT--
bool(false)
bool(false)
string(4) "gzip"
Enter fullscreen mode Exit fullscreen mode

Now run the test via:

$ make test TESTS=test/zlib_get_coding_type.phpt
Enter fullscreen mode Exit fullscreen mode

The test failed again. Looking into the zlib_get_coding_type.log file, you notice that the third call to the zlib_get_coding_type() function returned false and not the expected string gzip. It seems like PHP is not reacting to the HTTP header that you clearly set just one line before. Did we spot a bug in PHP?

The source code tells you: Open the ext/zlib/zlib.c source code file and search for the variable compression_coding as this is the variable that is evaluated in the function you are testing. You should find some matches, but there is one at the beginning of the file in the C function php_zlib_output_encoding which looks like the only place where something is assigned to the variable in question.

Analysing the source code, not understanding exactly what is going on, and finally asking for help on the PHP Community Chat reveals the following: All of your userspace variables and PHP user-accessible autoglobals ($_GET, $_SERVER, ...) are copy-on-write. Altering those from your script creates a copy for you to work on in userspace, effectively making the HTTP request headers immutable to the userspace. The PHP core continues to use the original, unaltered version.

Now that you know you can not alter or set the HTTP header from within your script to influence PHP’s behavior, this might look like a sad end for your tests code coverage.

Wait, there is more! I did not tell you about another section in the PHPT file format that might come in handy now: The ENV section. This section is used to give environment variables to your script and are passed to the PHP process that runs your test code. This means you can go on and that you have to create a test file per test case. Let’s create a new test file for the gzip case and name it zlib_get_coding_type_gzip.phpt.

--TEST--
zlib_get_coding_type() is gzip
--SKIPIF--
<?php
if (!extension_loaded("zlib")) print "skip";
?>
--ENV--
HTTP_ACCEPT_ENCODING=gzip
--FILE--
<?php
ini_set('zlib.output_compression', 'Off');
$off = zlib_get_coding_type();
ini_set('zlib.output_compression', 'On');
$gzip = zlib_get_coding_type();
var_dump($on);
var_dump($gzip);
?>
--EXPECT--
bool(false)
string(4) "gzip"
Enter fullscreen mode Exit fullscreen mode

Run the test and see it fails once more. You might have expected this by now. 😝

Let’s see what happened by looking into the zlib_get_coding_type_gzip.log file.

---- EXPECTED OUTPUT
bool(false)
string(4) "gzip"
---- ACTUAL OUTPUT
�1�0
    ���B�P�Txꆘ8`5ؑ������s�7a�ze��B+���ϑ�,����^���~�^�J1����ʶ`�\0�v@cm��}�ap�X}��'4�ͩqG�^w
---- FAILED
Enter fullscreen mode Exit fullscreen mode

Wow, this looks like some binary garbage, and yeah, this does not match the expected output.

Look at your test case. You told PHP that you accept gzip encoding and you turned on output compression. PHP is basically just doing what we told it to do. The binary garbage you see here is gzip encoded data. This means you succeeded in activating gzip output compression, but forgot to turn it off before dumping out the two variables. Now, all that’s left is to add another call to deactivate output compression again so the final test looks similar to this:

--TEST--
zlib_get_coding_type() is gzip
--SKIPIF--
<?php
if (!extension_loaded("zlib")) print "skip";
?>
--ENV--
HTTP_ACCEPT_ENCODING=gzip
--FILE--
<?php
ini_set('zlib.output_compression', 'Off');
$off = zlib_get_coding_type();
ini_set('zlib.output_compression', 'On');
$gzip = zlib_get_coding_type();
ini_set('zlib.output_compression', 'Off');
var_dump($off);
var_dump($gzip);
?>
--EXPECT--
bool(false)
string(4) "gzip"
Enter fullscreen mode Exit fullscreen mode

Run this test again, and it finally passes! 🎊🥳🎉

This leaves you with writing one test case for the Accept-Encoding header being set to deflate and another test case for the Accept-Encoding header being set to an invalid string (basically something that is not gzip or deflate). But this is just diligence from this point on and if you like to cheat you can find the tests in the ext/zlib/tests/ directory named zlib_get_coding_type_basic.phpt, zlib_get_coding_type_br.phpt, zlib_get_coding_type_deflate.phpt and zlib_get_coding_type_gzip.phpt.

Collect Some Evidence!

Now that you have tests for all four cases you could hope you covered that function with tests or (a way better option if you ask me), you could continue and create a code coverage report! For this to work you need to install the lcov tool via your Linux package manager (sudo dnf install lcov for Fedora or sudo apt-get install lcov for Debian). As PHP was built with gcov disabled, we need to run configure again, with enabled gcov and recompile.

$ make clean
$ CCACHE_DISABLE=1 ./configure --with-zlib --enable-gcov
$ make -j `nproc`
Enter fullscreen mode Exit fullscreen mode

This compiles the PHP binary with enabled code coverage generation. Lcov will be used to create an HTML code coverage report afterwards. To create your code coverage report, you need to rerun the tests.

$ make test TESTS=test/zlib_get_coding_type.phpt
Enter fullscreen mode Exit fullscreen mode

While running the tests, gcov generates coverage files with a gcda file ending for every source code file. To generate the HTML code coverage report, run the following:

$ make lcov
Enter fullscreen mode Exit fullscreen mode

You may find the resulting HTML code coverage report in lcov_html/index.html.

Code coverage report now shows 9 of 10 lines covered from the `zlib_get_coding_type()` function

It looks like we covered all four cases.

What’s in It for You?

With every test you write PHP becomes more stable and reliable, as functions may not change behavior suddenly between releases.

Writing tests for PHP gives you a deeper understanding of how your favorite language works internally. In fact, by reading up to this point you learned that HTTP request headers are immutable to userspace, that there is such a thing as userspace, and that there are handler functions called and check what you are doing when you want to change ini settings at runtime.

Also knowing the PHPT file format gives you other benefits as well: PHPUnit not only supports the PHPT file format and can execute those tests, part of PHPUnit is tested with PHPT test cases. This led me to become a contributor to PHPUnit: Writig tests for PHPUnit in the PHPT file format.

Shout Out!

I would like to thank everyone involved in the PHP TestFest, but especially Ben Ramsey and everyone else involved in the organization of the 2017 edition. This was what made me write tests for PHP and made me not only a PHP but also a PHPUnit contributor. In the long run, this brought me my first ElePHPant 🐘!

Closing notes

I would like to thank my proof readers Kara Ferguson and Pim Elsfhof. It was a pleasure working with you!

"Wooble" is the property of Sammy Kaye Powers and is used with permission.

The German version is available through the PHP Magazin

Interested in seeing the talk to this blog post? Nomad PHP has you covered.

Posted on by:

realflowcontrol profile

Florian Engelhardt

@realflowcontrol

Proud dad of five 🧒, husband, Linux 🐧, Vim, PHP 🐘, OpenSource, Geek and I'm getting shit done

Discussion

pic
Editor guide
 

I got stuck at libxml, tried to use --with-zlib but it didn't work for me

configure: error: Package requirements (libxml-2.0 >= 2.9.0) were not met:

Package 'libxml-2.0', required by 'virtual:world', not found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables LIBXML_CFLAGS
and LIBXML_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.
Enter fullscreen mode Exit fullscreen mode

Do I need to do something else specific to that?

Apart from it, it's a nice guide, and it has motivated me to try it out to potentially contribute in the future.

 

Hey Agustin,

this looks like the libxml header files are not installed.

apt-get install libxml2-dev # for Debian based
# or
yum install libxml2-devel # for Fedora based
Enter fullscreen mode Exit fullscreen mode

This should install the needed header files to compile PHP. Additionally I could find this Stack Overflow Thread on the topic.

Let me know if this helped you!

Happy coding
Florian

 

Isn't is build-essential instead of build-essentials on Debian systems ? at least it is on my Ubuntu.

 

Nice catch. I updated the article accordingly, thank you!

 

I haven't got it all working locally yet, will test it later. If I find other things/suggestions I'll let you know.

Awesome, that would be great! 👍
I am on Fedora Linux, but there might be other/additional challenges on other distributions.