In September 2000, I started my vocational training at an internet agency with two teams: one doing JavaServer Pages, and one was doing PHP. I was assigned to the PHP team, and when presented with the language, I immediately knew that no one will ever use this. I was wrong. Today, my entire career is built on PHP. It’s time to give back to the community by writing tests for the PHP core itself!
- gcov.php.net is retired, and code coverage reports can now be found at Azure Pipelines
Prepare Your machine!
Before you start writing tests for PHP, let’s start with running the tests that already exist. Fetch the PHP source from GitHub and compile it to do so.
$ git clone email@example.com:php/php-src.git $ cd php-src $ ./buildconf $ ./configure --with-zlib $ make -j `nproc`
I recommend creating a fork upfront because it makes creating a pull request with your test easier, later on.
If you do not have a compiler and build tools already installed on your Linux computer, you should install the
development-tools group on Fedora or the
build-essential on Debian Linux. The
./configure command may exit with an error condition; this usually occurs when build requirements are not met, so install whatever
./configure is missing and re-run that step. Keep in mind that you need to install the development packages — in my case, the
configure script stated it was missing
libxml, which was, in fact, installed. What it really missed was the header files, which are in the development package (usually named with a dev or devel suffix). Note that the
--with-zlib is mandatory in this case, as you need the zlib extension which is not built by default.
After your build is complete, you can find the PHP binary in
./sapi/cli/php. Go ahead and check what you just created:
$ ./sapi/cli/php –v PHP 8.0.0-dev (cli) (built: Jul 14 2020 21:05:42) ( NTS ) Copyright (c) The PHP Group Zend Engine v4.0.0-dev, Copyright (c) Zend Technologies
Now that you have a freshly built and running PHP binary, you can finally run the tests included within the GitHub repository. Since PHP 7.4 tests may run in parallel, you can give the number of parallel jobs with the
$ make TEST_PHP_ARGS=-j`nproc` test ... ============================================================= TEST RESULT SUMMARY ------------------------------------------------------------- Exts skipped : 46 Exts tested : 26 ------------------------------------------------------------- Number of tests : 15670 10668 Tests skipped : 5002 ( 31.9%) -------- Tests warned : 0 ( 0.0%) ( 0.0%) Tests failed : 0 ( 0.0%) ( 0.0%) Expected fail : 32 ( 0.2%) ( 0.3%) Tests passed : 10636 ( 67.9%) ( 99.7%) ------------------------------------------------------------- Time taken : 76 seconds =============================================================
This looks good and it only took 76 seconds to run 10636 tests — quite fast. Zero warnings and zero failures, hooray! You can see that 5002 tests have been skipped and 32 have been expected to fail. You will learn about why this is in the next part.
What Do Tests Look Like?
Let’s take a look at what such a test case may look like. PHP test files have the file ending
.phpt and consist of several sections, three of which are mandatory: the
FILE and the
--TEST-- strlen() function --FILE-- <?php var_dump(strlen('Hello World!')); ?> --EXPECT-- int(12)
The TEST Section
This is a short description of what you are testing. It should be as short as possible, as this is what is printed when running the tests. You can put additional details in the
The SKIPIF Section
This section is optional and is parsed by PHP. If the resulting output starts with the word
skip, the test case is skipped. If it starts with
xfail the test case is run, but it is expected to fail. The primary use case is to check whether a required extension is installed.
The FILE Section
This is the actual test case, PHP code enclosed by PHP tags. This is where you do whatever you want to test. As you have to create output that is then matched against your expectation, you usually find
var_dump all over the place.
The EXPECT Section
This section must exactly match the output from the executed code in the
FILE section to pass.
The EXPECTF Section
EXPECTF can be used as an alternative to the
EXPECT section and allows the usage of substitution tags you may know from the
printf functions family:
The XFAIL Section
XFAIL is another optional section. If present, the test case is expected to fail; you don’t need to echo out
xfail in the
SKIPIF section at all if you have this. It contains a description of why this test case is expected to fail. This feature is mainly used if the test is already finished, but the implementation isn’t yet, or for upstream bugs. It usually contains a link to where a discussion about this topic can be found.
The CLEAN Section
This exists so you can clean up after yourself. An example might be temporary files you created during the test. Keep in mind, this section's code is executed independently from the
FILES section, so you have no access to variables declared over there. Also this section is run regardless of the outcome of the test.
You may find more in depth details on the file format at the PHP Quality Assurance Team Web Page.
What Can You Test?
Now that you know what a PHP test looks like it is time to find something to test. Head over to Azure Pipelines, click on the latest scheduled run (look for the calendar icon and the string "Scheduled for") and then click “Code Coverage” to view the code coverage report.
When I started looking for something to test, I found that the
zlib_get_coding_type() function in
ext/zlib/zlib.c was not covered at all (this was back in the PHP 7.1 branch and on the now retired gcov.php.net).
The next step was to check out what this function was supposed to do in the PHP Documentation. What I saw in the documentation, but also in the code itself, was that this function returns the string
deflate or the Boolean value
false. The linked zlib.output_compression directive documentation gave one additional bit of information: the zlib output compression feature reacts on the HTTP
Accept-Encoding header sent with the client HTTP request.
For the test, this means there are four possible cases to check for:
- the absence of the Accept-Encoding header
- the Accept-Encoding being
- the Accept-Encoding being
- the Accept-Encoding being anything else
The last case is treated the same as the first case: The function is expected to return the Boolean value
Time To Write That Test!
Let's start with the first test in the file
test/zlib_get_coding_type.phpt for the case that there is no
Accept-Encoding header set.
--TEST-- zlib_get_coding_type() --SKIPIF-- <?php if (!extension_loaded("zlib")) print "skip"; ?> --FILE-- <?php ini_set('zlib.output_compression', 'Off'); var_dump(zlib_get_coding_type()); ini_set('zlib.output_compression', 'On'); var_dump(zlib_get_coding_type()); ?> --EXPECT-- bool(false) bool(false)
You can run this single test via:
$ make test TESTS=test/zlib_get_coding_type.phpt
Sadly, this gives you a failed test. You can easily see why this is the case, by checking the
test directory contents where you find your
.phpt test file and some other files with various file endings. One that is particularly interesting, in this case, is the
---- EXPECTED OUTPUT bool(false) bool(false) ---- ACTUAL OUTPUT bool(false) Warning: ini_set(): Cannot change zlib.output_compression - headers already sent in test/zlib_get_coding_type.php on line 4 bool(false) ---- FAILED
And this is your first time learning about PHP internals. You cannot change the output compression setting after your script created any form of output. For this to work, there seems to be a handler that is called when we change the
zlib.output_compression directive. Try searching for "zlib.output_compression" in the
ext/zlib/zlib.c source code file. You find it in the call to the
STD_PHP_INI_BOOLEAN macro along with the pointer to the function
OnUpdate_zlib_output_compression in which you can spot the warning you received. To work around this warning, you can change the
<?php ini_set('zlib.output_compression', 'Off'); $off = zlib_get_coding_type(); ini_set('zlib.output_compression', 'On'); $on = zlib_get_coding_type(); var_dump($off); var_dump($on); ?>
Rerunning the test case results in your first successful test! 🎉
It’s not time to party yet; there are still other cases to cover. The next one on the list is the case with the HTTP
Accept-Encoding header set to the string
gzip. As a PHP developer, you know you can access and change the HTTP headers via the super global
$_SERVER. Let’s change the
EXPECT section accordingly.
--FILE-- <?php ini_set('zlib.output_compression', 'Off'); $off = zlib_get_coding_type(); ini_set('zlib.output_compression', 'On'); $on = zlib_get_coding_type(); $_SERVER['HTTP_ACCEPT_ENCODING'] = 'gzip'; $gzip = zlib_get_coding_type(); var_dump($off); var_dump($on); var_dump($gzip); ?> --EXPECT-- bool(false) bool(false) string(4) "gzip"
Now run the test via:
$ make test TESTS=test/zlib_get_coding_type.phpt
The test failed again. Looking into the
zlib_get_coding_type.log file, you notice that the third call to the
zlib_get_coding_type() function returned
false and not the expected string
gzip. It seems like PHP is not reacting to the HTTP header that you clearly set just one line before. Did we spot a bug in PHP?
The source code tells you: Open the
ext/zlib/zlib.c source code file and search for the variable
compression_coding as this is the variable that is evaluated in the function you are testing. You should find some matches, but there is one at the beginning of the file in the C function
php_zlib_output_encoding which looks like the only place where something is assigned to the variable in question.
Analysing the source code, not understanding exactly what is going on, and finally asking for help on the PHP Community Chat reveals the following: All of your userspace variables and PHP user-accessible autoglobals (
$_SERVER, ...) are copy-on-write. Altering those from your script creates a copy for you to work on in userspace, effectively making the HTTP request headers immutable to the userspace. The PHP core continues to use the original, unaltered version.
Now that you know you can not alter or set the HTTP header from within your script to influence PHP’s behavior, this might look like a sad end for your tests code coverage.
Wait, there is more! I did not tell you about another section in the PHPT file format that might come in handy now: The
ENV section. This section is used to give environment variables to your script and are passed to the PHP process that runs your test code. This means you can go on and that you have to create a test file per test case. Let’s create a new test file for the gzip case and name it
--TEST-- zlib_get_coding_type() is gzip --SKIPIF-- <?php if (!extension_loaded("zlib")) print "skip"; ?> --ENV-- HTTP_ACCEPT_ENCODING=gzip --FILE-- <?php ini_set('zlib.output_compression', 'Off'); $off = zlib_get_coding_type(); ini_set('zlib.output_compression', 'On'); $gzip = zlib_get_coding_type(); var_dump($on); var_dump($gzip); ?> --EXPECT-- bool(false) string(4) "gzip"
Run the test and see it fails once more. You might have expected this by now. 😝
Let’s see what happened by looking into the
---- EXPECTED OUTPUT bool(false) string(4) "gzip" ---- ACTUAL OUTPUT �1�0 ���B�P�Txꆘ8`5ؑ������s�7a�ze��B+���ϑ�,����^���~�^�J1����ʶ`�\0�v@cm��}�ap�X}��'4�ͩqG�^w ---- FAILED
Wow, this looks like some binary garbage, and yeah, this does not match the expected output.
Look at your test case. You told PHP that you accept gzip encoding and you turned on output compression. PHP is basically just doing what we told it to do. The binary garbage you see here is gzip encoded data. This means you succeeded in activating gzip output compression, but forgot to turn it off before dumping out the two variables. Now, all that’s left is to add another call to deactivate output compression again so the final test looks similar to this:
--TEST-- zlib_get_coding_type() is gzip --SKIPIF-- <?php if (!extension_loaded("zlib")) print "skip"; ?> --ENV-- HTTP_ACCEPT_ENCODING=gzip --FILE-- <?php ini_set('zlib.output_compression', 'Off'); $off = zlib_get_coding_type(); ini_set('zlib.output_compression', 'On'); $gzip = zlib_get_coding_type(); ini_set('zlib.output_compression', 'Off'); var_dump($off); var_dump($gzip); ?> --EXPECT-- bool(false) string(4) "gzip"
Run this test again, and it finally passes! 🎊🥳🎉
This leaves you with writing one test case for the
Accept-Encoding header being set to
deflate and another test case for the
Accept-Encoding header being set to an invalid string (basically something that is not
deflate). But this is just diligence from this point on and if you like to cheat you can find the tests in the
ext/zlib/tests/ directory named
Collect Some Evidence!
Now that you have tests for all four cases you could hope you covered that function with tests or (a way better option if you ask me), you could continue and create a code coverage report! For this to work you need to install the
lcov tool via your Linux package manager (
sudo dnf install lcov for Fedora or
sudo apt-get install lcov for Debian). As PHP was built with
gcov disabled, we need to run configure again, with enabled
gcov and recompile.
$ make clean $ CCACHE_DISABLE=1 ./configure --with-zlib --enable-gcov $ make -j `nproc`
This compiles the PHP binary with enabled code coverage generation. Lcov will be used to create an HTML code coverage report afterwards. To create your code coverage report, you need to rerun the tests.
$ make test TESTS=test/zlib_get_coding_type.phpt
While running the tests,
gcov generates coverage files with a
gcda file ending for every source code file. To generate the HTML code coverage report, run the following:
$ make lcov
You may find the resulting HTML code coverage report in
It looks like we covered all four cases.
What’s in It for You?
With every test you write PHP becomes more stable and reliable, as functions may not change behavior suddenly between releases.
Writing tests for PHP gives you a deeper understanding of how your favorite language works internally. In fact, by reading up to this point you learned that HTTP request headers are immutable to userspace, that there is such a thing as userspace, and that there are handler functions called and check what you are doing when you want to change ini settings at runtime.
Also knowing the PHPT file format gives you other benefits as well: PHPUnit not only supports the PHPT file format and can execute those tests, part of PHPUnit is tested with PHPT test cases. This led me to become a contributor to PHPUnit: Writig tests for PHPUnit in the PHPT file format.
I would like to thank everyone involved in the PHP TestFest, but especially Ben Ramsey and everyone else involved in the organization of the 2017 edition. This was what made me write tests for PHP and made me not only a PHP but also a PHPUnit contributor. In the long run, this brought me my first ElePHPant 🐘!
I would like to thank my proof readers Kara Ferguson and Pim Elsfhof. It was a pleasure working with you!
"Wooble" is the property of Sammy Kaye Powers and is used with permission.
The German version is available through the PHP Magazin
Interested in seeing the talk to this blog post? Nomad PHP has you covered.
Top comments (6)
I got stuck at libxml, tried to use
--with-zlibbut it didn't work for me
Do I need to do something else specific to that?
Apart from it, it's a nice guide, and it has motivated me to try it out to potentially contribute in the future.
this looks like the
libxmlheader files are not installed.
This should install the needed header files to compile PHP. Additionally I could find this Stack Overflow Thread on the topic.
Let me know if this helped you!
build-essentialson Debian systems ? at least it is on my Ubuntu.
Nice catch. I updated the article accordingly, thank you!
I haven't got it all working locally yet, will test it later. If I find other things/suggestions I'll let you know.
Awesome, that would be great! 👍
I am on Fedora Linux, but there might be other/additional challenges on other distributions.