DEV Community: Florian Engelhardt

To double quote or not, that's the question!

Florian Engelhardt — Fri, 16 Aug 2024 07:54:09 +0000

Just recently I heard again that PHP folks still talk about single quotes vs. double quotes and that using single quotes is just a micro optimisation but if you get used to using single quotes all the time you'd save a bunch of CPU cycles!

"Everything has already been said, but not yet by everyone" – Karl Valentin

It is in this spirit that I am writing an article about the same topic Nikita Popov did already 12 years ago (if you are reading his article, you can stop reading here).

What is the fuzz all about?

PHP performs string interpolation, in which it searches for the use of variables in a string and replaces them with the value of the variable used:

$juice = "apple";
echo "They drank some $juice juice.";
// will output: They drank some apple juice.

This feature is limited to strings in double quotes and heredoc. Using single quotes (or nowdoc) will yield a different result:

$juice = "apple";
echo 'They drank some $juice juice.';
// will output: They drank some $juice juice.

Look at that: PHP will not search for variables in that single quoted string. So we could just start using single quotes everywhere. So people started suggesting changes like this ..

- $juice = "apple";
+ $juice = 'apple';

.. because it'll be faster and it'd save a bunch of CPU cycles with every execution of that code because PHP does not look for variables in single quoted strings (which are non-existent in the example anyway) and everyone is happy, case closed.

Case closed?

Obviously there is a difference in using single quotes vs. double quotes, but in order to understand what is going on we need to dig a bit deeper.

Even though PHP is an interpreted language it is using a compile step in which certain parts play together to get something the virtual machine can actually execute, which is opcodes. So how do we get from PHP source code to opcodes?

The lexer

The lexer scans the source code file and breaks it down into tokens. A simple example of what this means can be found in the token_get_all() function documentation. A PHP source code of just <?php echo ""; becomes these tokens:

T_OPEN_TAG (<?php )
T_ECHO (echo)
T_WHITESPACE ( )
T_CONSTANT_ENCAPSED_STRING ("")

We can see this in action and play with it in this 3v4l.org snippet.

The parser

The parser takes these tokens and generates an abstract syntax tree from them. An AST representation of the above example looks like this when represented as a JSON:

{
  "data": [
    {
      "nodeType": "Stmt_Echo",
      "attributes": {
        "startLine": 1,
        "startTokenPos": 1,
        "startFilePos": 6,
        "endLine": 1,
        "endTokenPos": 4,
        "endFilePos": 13
      },
      "exprs": [
        {
          "nodeType": "Scalar_String",
          "attributes": {
            "startLine": 1,
            "startTokenPos": 3,
            "startFilePos": 11,
            "endLine": 1,
            "endTokenPos": 3,
            "endFilePos": 12,
            "kind": 2,
            "rawValue": "\"\""
          },
          "value": ""
        }
      ]
    }
  ]
}

In case you wanna play with this as well and see how the AST for other code looks like, I found https://phpast.com/ by Ryan Chandler and https://php-ast-viewer.com/ which both show you the AST of a given piece of PHP code.

The compiler

The compiler takes the AST and creates opcodes. The opcodes are the things the virtual machine executes, it is also what will be stored in the OPcache if you have that setup and enabled (which I highly recommend).

To view the opcodes we have multiple options (maybe more, but I do know these three):

use the vulcan logic dumper extension. It is also baked into 3v4l.org
use phpdbg -p script.php to dump the opcodes
or use the opcache.opt_debug_level INI setting for OPcache to make it print out the opcodes
- a value of 0x10000 outputs opcodes before optimisation
- a value of 0x20000 outputs opcodes after optimisation

$ echo '<?php echo "";' > foo.php
$ php -dopcache.opt_debug_level=0x10000 foo.php
$_main:
...
0000 ECHO string("")
0001 RETURN int(1)

Hypothesis

Coming back to the initial idea of saving CPU cycles when using single quotes vs. double quotes, I think we all agree that this would only be true if PHP would evaluate these strings at runtime for every single request.

What happens at runtime?

So let's see which opcodes PHP creates for the two different versions.

Double quotes:

<?php echo "apple";

0000 ECHO string("apple")
0001 RETURN int(1)

vs. single quotes:

<?php echo 'apple';

0000 ECHO string("apple")
0001 RETURN int(1)

Hey wait, something weird happened. This looks identical! Where did my micro optimisation go?

Well maybe, just maybe the ECHO opcode handler's implementation parses the given string, although there is no marker or something else which tells it to do so ... hmm 🤔

Let's try a different approach and see what the lexer does for those two cases:

Double quotes:

T_OPEN_TAG (<?php )
T_ECHO (echo)
T_WHITESPACE ( )
T_CONSTANT_ENCAPSED_STRING ("")

vs. single quotes:

T_OPEN_TAG (<?php )
T_ECHO (echo)
T_WHITESPACE ( )
T_CONSTANT_ENCAPSED_STRING ('')

The tokens are still distinguishing between double and single quotes, but checking the AST will give us an identical result for both cases - the only difference is the rawValue in the Scalar_String node attributes, that still has the single/double quotes, but the value uses double quotes in both cases.

New Hypothesis

Could it be, that string interpolation is actually done at compile time?

Let's check with a slightly more "sophisticated" example:

<?php
$juice="apple";
echo "juice: $juice";

Tokens for this file are:

T_OPEN_TAG (<?php)
T_VARIABLE ($juice)
T_CONSTANT_ENCAPSED_STRING ("apple")
T_WHITESPACE ()
T_ECHO (echo)
T_WHITESPACE ( )
T_ENCAPSED_AND_WHITESPACE (juice: )
T_VARIABLE ($juice)

Look at the last two tokens! String interpolation is handled in the lexer and as such is a compile time thing and has nothing to do with runtime.

For completeness, let's have a look at the opcodes generated by this (after optimisation, using 0x20000):

0000 ASSIGN CV0($juice) string("apple")
0001 T2 = FAST_CONCAT string("juice: ") CV0($juice)
0002 ECHO T2
0003 RETURN int(1)

This is different opcode than we had in our simple <?php echo ""; example, but this is okay because we are doing something different here.

Get to the point: should I concat or interpolate?

Let's have a look at these three different versions:

<?php
$juice = "apple";
echo "juice: $juice $juice";
echo "juice: ", $juice, " ", $juice;
echo "juice: ".$juice." ".$juice;

the first version is using string interpolation
the second is using a comma separation (which AFAIK only works with echo and not with assigning variables or anything else)
and the third option uses string concatenation

The first opcode assigns the string "apple" to the variable $juice:

0000 ASSIGN CV0($juice) string("apple")

The first version (string interpolation) is using a rope as the underlying data structure, which is optimised to do as little string copies as possible.

0001 T2 = ROPE_INIT 4 string("juice: ")
0002 T2 = ROPE_ADD 1 T2 CV0($juice)
0003 T2 = ROPE_ADD 2 T2 string(" ")
0004 T1 = ROPE_END 3 T2 CV0($juice)
0005 ECHO T1

The second version is the most memory effective as it does not create an intermediate string representation. Instead it does multiple calls to ECHO which is a blocking call from an I/O perspective so depending on your use case this might be a downside.

0006 ECHO string("juice: ")
0007 ECHO CV0($juice)
0008 ECHO string(" ")
0009 ECHO CV0($juice)

The third version uses CONCAT/FAST_CONCAT to create an intermediate string representation and as such might do more memory copies and/or use more memory than the rope version.

0010 T1 = CONCAT string("juice: ") CV0($juice)
0011 T2 = FAST_CONCAT T1 string(" ")
0012 T1 = CONCAT T2 CV0($juice)
0013 ECHO T1

So ... what is the right thing to do here and why is it string interpolation?

String interpolation uses either a FAST_CONCAT in the case of echo "juice: $juice"; or highly optimised ROPE_* opcodes in the case of echo "juice: $juice $juice";, but most important it communicates the intent clearly and none of this has been bottle neck in any of the PHP applications I have worked with so far, so none of this actually matters.

TLDR

String interpolation is a compile time thing. Granted, without OPcache the lexer will have to check for variables used in double quoted strings on every request, even if there aren't any, waisting CPU cycles, but honestly: The problem is not the double quoted strings, but not using OPcache!

However, there is one caveat: PHP up to 4 (and I believe even including 5.0 and maybe even 5.1, I don't know) did string interpolation at runtime, so using these versions ... hmm, I guess if anyone really still uses PHP 5, the same as above applies: The problem is not the double quoted strings, but the use of an outdated PHP version.

Final advice

Update to the latest PHP version, enable OPcache and live happily ever after!

[Edit: August 16th]

What about `sprintf()`?

So actually I intended to say that none of this is a performance problem, if you are using string interpolation, single quotes and concatenation or anything else. Someone stepped up and mentioned sprintf() and where this clocks in performance-wise. So for the sake of completeness, lets have a look at sprintf():

<?php
$juice = "apple";
echo sprintf("juice: %s %s", $juice, $juice);

compiles to the following opcode:

0000 ASSIGN CV0($juice) string("apple")
0001 INIT_FCALL 3 128 string("sprintf")
0002 SEND_VAL string("juice: %s %s") 1
0003 SEND_VAR CV0($juice) 2
0004 SEND_VAR CV0($juice) 3
0005 V1 = DO_ICALL
0006 ECHO V1

A quick benchmark shows that the sprintf() variant takes 14 to 21 times as long as the string interpolation variant on my local machine.

Here comes the catch: this is only true up to PHP 8.3, PHP 8.4 comes with another compile time optimisation that will treat sprintf() calls that just have %s and %d in them as if you wrote string interpolation:

0000 ASSIGN CV0($juice) string("apple")
0001 T2 = ROPE_INIT 4 string("juice: ")
0002 T2 = ROPE_ADD 1 T2 CV0($juice)
0003 T2 = ROPE_ADD 2 T2 string(" ")
0004 T1 = ROPE_END 3 T2 CV0($juice)
0005 ECHO T1

So the final advice still holds: update to the latest PHP version (well, maybe wait with upgrading to PHP 8.4 until there is a stable release).

Processing One Billion Rows in PHP!

Florian Engelhardt — Fri, 08 Mar 2024 09:27:36 +0000

You may have heard of the "The One Billion Row Challenge" (1brc) and in case you don't, go checkout Gunnar Morlings's 1brc repo.

I got sucked in because two of my colleagues have entered the competition and are on the leaderboard.

PHP is not known for its speed, but as I am working on the PHP profiler I thought I give it a shot and see how fast it can get.

A first naive approach

I cloned the repository and created the billion row dataset in measurements.txt. After that I started building my first naive implementation of something that could solve the challenge:

<?php

$stations = [];

$fp = fopen('measurements.txt', 'r');

while ($data = fgetcsv($fp, null, ';')) {
    if (!isset($stations[$data[0]])) {
        $stations[$data[0]] = [
            $data[1],
            $data[1],
            $data[1],
            1
        ];
    } else {
        $stations[$data[0]][3] ++;
        $stations[$data[0]][2] += $data[1];
        if ($data[1] < $stations[$data[0]][0]) {
            $stations[$data[0]][0] = $data[1];
        }
        if ($data[1] > $stations[$data[0]][1]) {
            $stations[$data[0]][1] = $data[1];
        }
    }
}

ksort($stations);

echo '{';
foreach($stations as $k=>&$station) {
    $station[2] = $station[2]/$station[3];
    echo $k, '=', $station[0], '/', $station[2], '/', $station[1], ', ';
}
echo '}';

There is nothing wild in here, just open the file, use fgetcsv() to read the data. If the station is not found already, create it, otherwise increment the counter, sum the temperature and see if the current temperature is lower than or higher than min or max and updated accordingly.

Once I have everything together, I use ksort() to bring the $stations array in order and then echo out the list and calculate the average temperature while doing so (sum / count).

Running this simple code on my laptop takes 25 Minutes 🤯

Time to optimise and have a look at the profiler:

The timeline visualisation helps me see, that this is clearly CPU bound, file compilation at the beginning of the script is negligible and there are no garbage collection events.

The flame graph view is helpful as well in showing that I am spending 46% of CPU time in fgetcsv().

`fgets()` instead of `fgetcsv()`

The first optimisation was to use fgets() to get a line and split on the ; character manually instead of relying on fgetcsv(). This is because fgetcsv() does a whole lot more than what I need.

// ...
while ($data = fgets($fp, 999)) {
    $pos = strpos($data, ';');
    $city = substr($data, 0, $pos);
    $temp = substr($data, $pos+1, -1);
// ...

Additionally I refactored $data[0] to $city and $data[1] to $temp everywhere.

Running the script again with just this one change already brought the runtime down to 19m 49s. In absolute numbers, this is still a lot but also: it's 21% down!

The flame graph reflects the change, switching to showing the CPU time by line also reveals what is going on in the root frame:

I am spending ~38% CPU time in lines 18 and 23, which are:

18 | $stations[$city][3] ++;
   | // ...
23 | if ($temp > $stations[$city][1]) {

Line 18 is the first access to the $stations array in the loop, otherwise it is only an increment and line 23 is a comparison, nothing that seems expensive at first glance, but lets do few more optimisations and you'll see what is taking time here.

Using a reference where possible

$station = &$stations[$city];
$station[3] ++;
$station[2] += $temp;
// instead of
$stations[$city][3] ++;
$stations[$city][2] += $data[1];

This should help PHP to not search for the key in the $stations array on every array access, see it like a cache for accessing the "current" station in the array.

And it actually helps, running this only takes 17m 48s, another 10% down!

Only one comparison

While looking at the code, I stumbled over this piece:

if ($temp < $station[0]) {
    $station[0] = $temp;
}
if ($temp > $station[1]) {
    $station[1] = $temp;
}

In case the temperature is lower than min, it cannot be higher than max anymore, so I made this an elseif and maybe spare some CPU cycles.

BTW: I don't know anything about the order of temperatures in measurements.txt, but depending on that order it could make a difference if I first checked the one or the other.

The new versions takes 17m 30s, which is another ~2%. Better than just jitter, but not by a lot.

Adding type casts

PHP is known as a dynamic language and it is something that I valued a lot when I just got started writing software, one less problem to care about. But on the other side, knowing the types helps the engine make better decisions when running your code.

$temp = (float)substr($data, $pos+1, -1);

Guess what? This simple cast makes the script run in just 13m 32s which is a whopping performance increase of 21%!

18 | $station = &$stations[$city];
   | // ...
23 | } elseif ($temp > $station[1]) {

Line 18 still shows up with 11% of CPU time spend, which is the access to the array (finding the key in the hash map, which is the underling data structure used for associative arrays in PHP).

Line 23's CPU time dropped from ~32% to ~15%. This is due to PHP not doing type juggling anymore. Before the type cast, $temp / $station[0] / $station[1] were strings, so PHP had to convert them to float in order to compare them on every single comparison.

What about JIT?

OPCache in PHP is by default disabled in CLI and needs the opcache.enable_cli setting to be set to on. JIT (as part of OPCache) is default enabled, but effectively disabled as the buffer size is set to 0, so I set the opcache.jit-buffer-size to something, I just went with 10M. After these changes have been applied, I re-ran the script with JIT and see it finish in:

7m 19s 🚀

which is 45.9% less time spend!!

What more?

I already brought the runtime down from 25 minutes in the beginning to just ~7 minutes. One thing that I found absolutely astonishing is that fgets() allocates ~56 GiB/m of RAM for reading a 13 GB file. Something seems off, so I checked the implementation of fgets() and it looks like that I can spare a lot of these allocations by just omitting the len argument to fgets():

while ($data = fgets($fp)) {
// instead of
while ($data = fgets($fp, 999)) {

Comparing the profile before and after the change gives this:

You might think that this gives a lot of performance improvement, but it is just ~1%. This is because those are small allocations which the ZendMM can handle in bins and those are blazingly fast.

Can we make it even faster?

Yes, we can! So far my approach was single threaded, which is the nature of most PHP software, but PHP does support threads in user land through the parallel extension.

Reading the data in PHP is a bottleneck as the profiler clearly shows. Switching from fgetcsv() to fgets() and manual split helps, but this is still taking a lot of time, so lets use threads to parallelise reading and processing the data and then afterwards combine the intermediate results from the worker threads.

<?php

$file = 'measurements.txt';

$threads_cnt = 16;

/**
 * Get the chunks that each thread needs to process with start and end position.
 * These positions are aligned to \n chars because we use `fgets()` to read
 * which itself reads till a \n character.
 *
 * @return array<int, array{0: int, 1: int}>
 */
function get_file_chunks(string $file, int $cpu_count): array {
    $size = filesize($file);

    if ($cpu_count == 1) {
        $chunk_size = $size;
    } else {
        $chunk_size = (int) ($size / $cpu_count);
    }

    $fp = fopen($file, 'rb');

    $chunks = [];
    $chunk_start = 0;
    while ($chunk_start < $size) {
        $chunk_end = min($size, $chunk_start + $chunk_size);

        if ($chunk_end < $size) {
            fseek($fp, $chunk_end);
            fgets($fp); // moves fp to next \n char
            $chunk_end = ftell($fp);
        }

        $chunks[] = [
            $chunk_start,
            $chunk_end
        ];

        $chunk_start = $chunk_end;
    }

    fclose($fp);
    return $chunks;
}

/**
 * This function will open the file passed in `$file` and read and process the
 * data from `$chunk_start` to `$chunk_end`.
 *
 * The returned array has the name of the city as the key and an array as the
 * value, containing the min temp in key 0, the max temp in key 1, the sum of
 * all temperatures in key 2 and count of temperatures in key 3.
 *
 * @return array<string, array{0: float, 1: float, 2: float, 3: int}>
 */ 
$process_chunk = function (string $file, int $chunk_start, int $chunk_end): array {
    $stations = [];
    $fp = fopen($file, 'rb');
    fseek($fp, $chunk_start);
    while ($data = fgets($fp)) {
        $chunk_start += strlen($data);
        if ($chunk_start > $chunk_end) {
            break;
        }
        $pos2 = strpos($data, ';');
        $city = substr($data, 0, $pos2);
        $temp = (float)substr($data, $pos2+1, -1);
        if (isset($stations[$city])) {
            $station = &$stations[$city];
            $station[3] ++;
            $station[2] += $temp;
            if ($temp < $station[0]) {
                $station[0] = $temp;
            } elseif ($temp > $station[1]) {
                $station[1] = $temp;
            }
        } else {
            $stations[$city] = [
                $temp,
                $temp,
                $temp,
                1
            ];
        }
    }
    return $stations;
};

$chunks = get_file_chunks($file, $threads_cnt);

$futures = [];

for ($i = 0; $i < $threads_cnt; $i++) {
    $runtime = new \parallel\Runtime();
    $futures[$i] = $runtime->run(
        $process_chunk,
        [
            $file,
            $chunks[$i][0],
            $chunks[$i][1]
        ]
    );
}

$results = [];

for ($i = 0; $i < $threads_cnt; $i++) {
    // `value()` blocks until a result is available, so the main thread waits
    // for the thread to finish
    $chunk_result = $futures[$i]->value();
    foreach ($chunk_result as $city => $measurement) {
        if (isset($results[$city])) {
            $result = &$results[$city];
            $result[2] += $measurement[2];
            $result[3] += $measurement[3];
            if ($measurement[0] < $result[0]) {
                $result[0] = $measurement[0];
            }
            if ($measurement[1] < $result[1]) {
                $result[1] = $measurement[1];
            }
        } else {
            $results[$city] = $measurement;
        }
    }
}

ksort($results);

echo '{', PHP_EOL;
foreach($results as $k=>&$station) {
    echo "\t", $k, '=', $station[0], '/', ($station[2]/$station[3]), '/', $station[1], ',', PHP_EOL;
}
echo '}', PHP_EOL;

This code does a few things, first I scan the file and split it into chunks that are \n aligned (for I am using fgets() later). When I have the chunks ready, I start $threads_cnt worker threads that then all open the same file and seek to their assigned chunk start and read and process the data till the chunk end, returning an intermediate result that afterwards gets combined, sorted and printed out in the main thread.

This multithreaded approach finishes in just:

1m 35s 🚀

Is this the end?

Nope, certainly not. There is at least two more things to this solution:

I am running this code on MacOS on Apple Silicon hardware which is crashing when using the JIT in a ZTS build of PHP, so the 1m 35s result is without JIT, it might be even faster if I could use it
I realised that I was running on a PHP version that was compiled using CFLAGS="-g -O0 ..." due my needs in my day to day work 🤦

I should have checked this in the beginning, so I re-compiled PHP 8.3 using CFLAGS="-Os ..." and my final number (with 16 threads) is:

🚀 27.7 seconds 🚀

This number is by no means comparable to what you can see in the leaderboard of the original challenge and this is due to the fact that I ran this code on total different hardware.

This is a timeline view with 10 threads:

The thread at the bottom is the main thread, waiting for the results from the worker threads. Once those workers have returned their intermediate results the main thread can be seen working on combining and sorting everything. We can also clearly see, that the main thread is by no means the bottleneck. In case you want to try to optimise this even further, concentrate on the worker threads.

What did I learn on the way?

Each abstraction layer simply trades usability/integration for CPU cycles or memory. fgetcsv() is super easy to use and hides a lot of stuff, but this comes at a cost. Even fgets() hides some stuff from us but makes it super convenient to read the data.

Adding types to your code will help the language either optimise execution or will stop type juggling which is something you do not see but still pay for it with CPU cycles.

JIT is awesome, especially when dealing with CPU bound problems!

It is definitely not the nature of most PHP software, but thanks to parallelisation (using ext-parallel) we could push the numbers down significantly.

The END?

I hope you had as much fun reading this article as I had. In case you want to further optimise the code, feel free to grab this and leave a comment how far you got.

[Edit to add the following on March, 18 2024]

There is MOAR!! performance to gain

After this blog post was released, some ideas arose from the comments how to make it faster and there were three suggestions from @xjakub

Remove the `isset()`

We might not need to check for isset(), we can "just" create the reference to the station, it will be NULL when the station does not exist. This means: one array access spared in case the city does exist (which is the majority of cases).

# before
if (isset($stations[$city])) {
    $station = &$stations[$city];
// ..

# after
$station = &$stations[$city];
if ($station !== NULL) {
// ..

This shaves off around 2.5% of wall time!

Don't check on the `fgets()` return value

The main loop that reads the file currently looks like this:

while ($data = fgets($fp)) {
    $chunk_start += strlen($data);
    if ($chunk_start > $chunk_end) {
        break;
    }
// ..

The added check for $chunk_start > $chunk_end came with moving to parallelisation, as every thread only works on a part of the file. Now @xjakub mentioned that there is no need to check for the fgets() return value anymore, as it will always return a string as long as we are in between $chunk_start and $chunk_end, meaning we could make this the expression in the loop and just read unchecked.

while ($chunk_start < $chunk_end) {
    $data = fgets($fp);
    $chunk_start += strlen($data);
// ..

This changes removes a branch from the loop and results in another ~2.7% drop in wall time!

`fgets()` vs. `stream_get_line()`

The actual reading and storing in $city and $temp looks as followes:

$data = fgets($fp);
$chunk_start += strlen($data);
$pos2 = strpos($data, ';');
$city = substr($data, 0, $pos2);
$temp = (float)substr($data, $pos2+1, -1);

I had never heard of stream_get_line() which behaves nearly identical to fgets() besides that it allows you to specify the end of line delimiter!

$city = stream_get_line($fp, 99, ';');
$temp = stream_get_line($fp, 99, "\n");
$chunk_start += strlen($city) + strlen($temp) + 2;
$temp = (float)$temp;

This change shoved off another ~15% of wall time!

Why is that? The implementations for fgets() and stream_get_line() are really close, both are using the PHP stream layer. The major thing that changed is that we do not need to split a string (from fgets()) into substrings using substr() anymore. The additional strlen() call is negligible as variables in PHP are zval's under the hood which hold the length of the string.

So where are we with the wall time for this PHP script?

@codemunky showed up in the comments and ran the benchmark on the same AX161 from Hetzner that the Java folks used to run their implementations on.

The final result (so far): 12.76 seconds 🎉

The END again?

I don't know, maybe there's still something to optimize here, but after managing to spend ~83% of the wall time in the stream_get_line() function, it looks like we've achieved what the PHP stream layer allows.

Either we find another function that bypasses the PHP stream layer and gives more direct access to the filesystem or we try to optimise the layer itself.

Happy hacking!

Revisiting GitLab as a PHP Developer

Florian Engelhardt — Thu, 18 Aug 2022 08:54:00 +0000

Every now and then it seems like a good idea to re-read the documentation of tools you've been using for years. In most cases you just update to the latest version and everything works fine, but you may not be using the latest features.

So here are some things I already knew, and some things I newly discovered when I re-read GitLab's documentation after stumbling over this tweet:

// Detect dark theme var iframe = document.getElementById('tweet-1461648425884303364-208'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1461648425884303364&theme=dark" }

TLDR

You may see all of the below in action in this demo merge request.

Unit test reports

The first thing I found is that GitLab has support for the JUnit XML file format and PHPUnit is able to write a JUnit XML file report. By configuring PHPUnit to create a JUnit XML and by declaring this an artefact in .gitlab-ci.yml you can have a nice unit test overview in the pipeline itself and the merge request.

Add the JUnit logging to your phpunit.xml file:

<phpunit>
    <!-- ... -->
    <logging>
        <junit outputFile="junit.xml" />
    </logging>
    <!-- ... -->
</phpunuit>

And tell GitLab-CI via the .gitlab-ci.yml about this report:

Unit Tests:
  stage: test
  script:
    - ./vendor/bin/phpunit
  artifacts:
    reports:
      junit: junit.xml

What you get is a nice overview of the tests run, the time it took to run every single one in the pipeline and in the merge request overview page.

You may find more hints about this feature in the unit test reports section of the official GitLab docs.

Test code coverage

GitLab can show you the code coverage in percent you have with your tests at various places:

as a badge via a predefined URL
in a merge request, also indicating if merging this code would increase or decrease code coverage
and as a nice graph over time in the Analytics > Repository overview per project and on a group level

Add a simple coverage report to phpunit.xml:

<phpunit>
    <!-- ... -->
    <coverage>
        <report>
            <text outputFile="php://stdout"
                  showUncoveredFiles="false"
                  showOnlySummary="true" />
        </report>
    </coverage>
</phpunuit>

And tell GitLab-CI via the .gitlab-ci.yml about this report (add the coverage key):

Unit Tests:
  stage: test
  script:
    - XDEBUG_MODE=coverage ./vendor/bin/phpunit
  coverage: '/^\s*Lines:\s*\d+.\d+\%./'

This assumes you are using Xdebug and have it already installed in the image your are using to run the tests in GitLab CI. Find the documentation about this feature here.

Test code coverage visualization

This one was an absolut blast for me. While you are gathering code coverage to have a line coverage number you can see everywhere in GitLab, you can also create a line coverage report as Cobertura XML file format with PHPUnit that GitLab can read and show you covered and uncovered lines directly in the merge requests diff view.

Add a the cobertura coverage report to phpunit.xml:

<phpunit>
    <!-- ... -->
    <coverage>
        <!-- ... -->
        <report>
            <!-- ... -->
            <cobertura outputFile="cobertura.xml" />
        </report>
    </coverage>
    <!-- ... -->
</phpunuit>

And tell GitLab-CI via the .gitlab-ci.yml about this report (add the coverage_report artifacts key):

Unit Tests:
  stage: test
  script:
    - XDEBUG_MODE=coverage ./vendor/bin/phpunit
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: cobertura.xml

Read more about this awesome feature in the test coverage visualization documentation alongside some screenshots.

Code style violations

PHP CS Fixer is able to generate a GitLab formatted report which GitLab can use to show you any new violations in the code or even if code style violations have been tackled.

Run PHP CS Fixer with the --format=gitlab argument and tell GitLab CI via the .gitlab-ci.yml where the report is to be found:

Coding guidelines:
  stage: static analysis
  script:
    - ./vendor/bin/php-cs-fixer fix -v --dry-run --format=gitlab --using-cache=no src/ > gl-cs-fixer.json || exit 0
  artifacts:
    reports:
      codequality: gl-cs-fixer.json

I hope this small overview helps you get more out of GitLab and GitLab CI when working with PHP projects.

Growing the PHP Core—One Test at a Time

Florian Engelhardt — Tue, 22 Sep 2020 08:58:27 +0000

In September 2000, I started my vocational training at an internet agency with two teams: one doing JavaServer Pages, and one was doing PHP. I was assigned to the PHP team, and when presented with the language, I immediately knew that no one will ever use this. I was wrong. Today, my entire career is built on PHP. It’s time to give back to the community by writing tests for the PHP core itself!

Prepare Your machine!

Before you start writing tests for PHP, let’s start with running the tests that already exist. Fetch the PHP source from GitHub and compile it to do so.

$ git clone git@github.com:php/php-src.git
$ cd php-src
$ ./buildconf
$ ./configure --with-zlib
$ make -j `nproc`

I recommend creating a fork upfront because it makes creating a pull request with your test easier, later on.

If you do not have a compiler and build tools already installed on your Linux computer, you should install the development-tools group on Fedora or the build-essential on Debian Linux. The ./configure command may exit with an error condition; this usually occurs when build requirements are not met, so install whatever ./configure is missing and re-run that step. Keep in mind that you need to install the development packages — in my case, the configure script stated it was missing libxml, which was, in fact, installed. What it really missed was the header files, which are in the development package (usually named with a dev or devel suffix). Note that the --with-zlib is mandatory in this case, as you need the zlib extension which is not built by default.

After your build is complete, you can find the PHP binary in ./sapi/cli/php. Go ahead and check what you just created:

$ ./sapi/cli/php –v
PHP 8.4.0-dev (cli) (built: Feb 21 2024 16:29:11) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.4.0-dev, Copyright (c) Zend Technologies

Now that you have a freshly built and running PHP binary, you can finally run the tests included within the GitHub repository. Since PHP 7.4 tests may run in parallel, you can give the number of parallel jobs with the -j argument.

$ make TEST_PHP_ARGS=-j`nproc` test
...
=============================================================
TEST RESULT SUMMARY
-------------------------------------------------------------
Exts skipped    :   46
Exts tested     :   26
-------------------------------------------------------------
Number of tests : 15670             10668
Tests skipped   : 5002 ( 31.9%) --------
Tests warned    :    0 (  0.0%) (  0.0%)
Tests failed    :    0 (  0.0%) (  0.0%)
Expected fail   :   32 (  0.2%) (  0.3%)
Tests passed    : 10636 ( 67.9%) ( 99.7%)
-------------------------------------------------------------
Time taken      :   76 seconds
=============================================================

This looks good and it only took 76 seconds to run 10636 tests — quite fast. Zero warnings and zero failures, hooray! You can see that 5002 tests have been skipped and 32 have been expected to fail. You will learn about why this is in the next part.

What Do Tests Look Like?

Let’s take a look at what such a test case may look like. PHP test files have the file ending .phpt and consist of several sections, three of which are mandatory: the TEST, the FILE and the EXPECT or EXPECTF section.

--TEST--
strlen() function
--FILE--
<?php
var_dump(strlen('Hello World!'));
?>
--EXPECT--
int(12)

The TEST Section

This is a short description of what you are testing. It should be as short as possible, as this is what is printed when running the tests. You can put additional details in the DESCRIPTION section.

The SKIPIF Section

This section is optional and is executed by the PHP binary under test. If the resulting output starts with the word skip, the test case is skipped. If it starts with xfail the test case is run, but it is expected to fail.

The FILE Section

This is the actual test case, PHP code enclosed by PHP tags. This is where you do whatever you want to test. As you have to create output that is then matched against your expectation, you usually find var_dump all over the place.

The EXPECT Section

This section must exactly match the output from the executed code in the FILE section to pass.

The EXPECTF Section

EXPECTF can be used as an alternative to the EXPECT section and allows the usage of substitution tags you may know from the printf functions family:

--EXPECTF--
int(%d)

The XFAIL Section

XFAIL is another optional section. If present, the test case is expected to fail; you don’t need to echo out xfail in the SKIPIF section at all if you have this. It contains a description of why this test case is expected to fail. This feature is mainly used if the test is already finished, but the implementation isn’t yet, or for upstream bugs. It usually contains a link to where a discussion about this topic can be found.

The CLEAN Section

This exists so you can clean up after yourself. An example might be temporary files you created during the test. Keep in mind, this section's code is executed independently from the FILES section, so you have no access to variables declared over there. Also this section is run regardless of the outcome of the test.

What else?

You may find more in depth details on the file format at the PHP Quality Assurance Team Web Page.

What Can You Test?

Now that you know what a PHP test looks like it is time to find something to test. Head over to codecov.io.

When I started looking for something to test, I found that the zlib_get_coding_type() function in ext/zlib/zlib.c was not covered at all (this was back in the PHP 7.1 branch and on the now retired gcov.php.net).

The next step was to check out what this function was supposed to do in the PHP Documentation. What I saw in the documentation, but also in the code itself, was that this function returns the string gzip or deflate or the Boolean value false. The linked zlib.output_compression directive documentation gave one additional bit of information: the zlib output compression feature reacts on the HTTP Accept-Encoding header sent with the client HTTP request.

For the test, this means there are four possible cases to check for:

the absence of the Accept-Encoding header
the Accept-Encoding being gzip
the Accept-Encoding being deflate
the Accept-Encoding being anything else

The last case is treated the same as the first case: The function is expected to return the Boolean value false.

Time To Write That Test!

Let's start with the first test in the file test/zlib_get_coding_type.phpt for the case that there is a Accept-Encoding header set to gzip. One question that might arise now is: How do I add an HTTP-Request in the tests? The answer is: you don't, but we know that HTTP-Request headers will be exposed via the $_SERVER super global, so we can inject the fake HTTP header there.

--TEST--
zlib_get_coding_type()
--EXTENSIONS--
zlib
--FILE--
<?php
$_SERVER["HTTP_ACCEPT_ENCODING"] = "gzip";
ini_set('zlib.output_compression', 'Off');
var_dump(zlib_get_coding_type());
ini_set('zlib.output_compression', 'On');
var_dump(zlib_get_coding_type());
?>
--EXPECT--
bool(false)
string(4) "gzip"

You can run this single test via:

$ make test TESTS=test/zlib_get_coding_type.phpt

Sadly, this gives you a failed test. You can easily see why this is the case, by checking the test directory contents where you find your .phpt test file and some other files with various file endings. One that is particularly interesting, in this case, is the zlib_get_coding_type.log file:

---- EXPECTED OUTPUT
bool(false)
string(4) "gzip"
---- ACTUAL OUTPUT
bool(false)
Warning: ini_set(): Cannot change zlib.output_compression - headers already sent in test/zlib_get_coding_type.php on line 4
bool(false)
---- FAILED

And this is your first time learning about PHP internals. You cannot change the output compression setting after your script created any form of output. For this to work, there seems to be a handler that is called when we change the zlib.output_compression directive. Try searching for "zlib.output_compression" in the ext/zlib/zlib.c source code file. You find it in the call to the STD_PHP_INI_BOOLEAN macro along with the pointer to the function OnUpdate_zlib_output_compression in which you can spot the warning you received. To work around this warning, you can change the FILE section:

<?php
$_SERVER["HTTP_ACCEPT_ENCODING"] = "gzip";
ini_set('zlib.output_compression', 'Off');
$off = zlib_get_coding_type();
ini_set('zlib.output_compression', 'On');
$on = zlib_get_coding_type();
var_dump($off, $on);
?>

Rerunning the test case results in yet another failed test. But this time you know where to look and you will find this:

---- EXPECTED OUTPUT
bool(false)
string(4) "gzip"
---- ACTUAL OUTPUT
bool(false)
bool(false)
---- FAILED

The warning is gone, but test failed again. That's progress!

Looking into the zlib_get_coding_type.log file, you notice that the second call to the zlib_get_coding_type() function returned false and not the expected string gzip. It seems like PHP is not reacting to the HTTP header that you clearly set just few lines before. Did we spot a bug in PHP?

The source code tells you: Open the ext/zlib/zlib.c source code file and search for the variable compression_coding as this is the variable that is evaluated in the function you are testing. You should find some matches, but there is one at the beginning of the file in the C function php_zlib_output_encoding which looks like the only place where something is assigned to the variable in question.

Analysing the source code, not understanding exactly what is going on, and finally asking for help on the PHP Community Chat reveals the following: All of your userspace variables and PHP user-accessible autoglobals ($_GET, $_SERVER, ...) are copy-on-write. Altering those from your script creates a copy for you to work on in userspace, effectively making the HTTP request headers immutable to the userspace. The PHP core continues to use the original, unaltered version.

Now that you know you can not alter or set the HTTP header from within your script to influence PHP’s behavior, this might look like a sad end for your tests code coverage.

Wait, there is more! I did not tell you about another section in the PHPT file format that might come in handy now: The ENV section. This section is used to give environment variables to your script and are passed to the PHP process that runs your test code. This means you can go on and that you have to create a test file per test case. Let’s create a new test file for the gzip case and name it zlib_get_coding_type_gzip.phpt.

--TEST--
zlib_get_coding_type() is gzip
--EXTENSIONS--
zlib
--ENV--
HTTP_ACCEPT_ENCODING=gzip
--FILE--
<?php
ini_set('zlib.output_compression', 'Off');
$off = zlib_get_coding_type();
ini_set('zlib.output_compression', 'On');
$gzip = zlib_get_coding_type();
var_dump($off);
var_dump($gzip);
?>
--EXPECT--
bool(false)
string(4) "gzip"

Run the test and see it fails once more. You might have expected this by now. 😝

Let’s see what happened by looking into the zlib_get_coding_type_gzip.log file.

---- EXPECTED OUTPUT
bool(false)
string(4) "gzip"
---- ACTUAL OUTPUT
�1�0
    ���B�P�Txꆘ8`5ؑ������s�7a�ze��B+���ϑ�,����^���~�^�J1����ʶ`�\0�v@cm��}�ap�X}��'4�ͩqG�^w
---- FAILED

Wow, this looks like some binary garbage, and yeah, this does not match the expected output.

Look at your test case. You told PHP that you accept gzip encoding and you turned on output compression. PHP is basically just doing what we told it to do. The binary garbage you see here is gzip encoded data. This means you succeeded in activating gzip output compression, but forgot to turn it off before dumping out the two variables. Now, all that’s left is to add another call to deactivate output compression again so the final test looks similar to this:

--TEST--
zlib_get_coding_type() is gzip
--EXTENSIONS--
zlib
--ENV--
HTTP_ACCEPT_ENCODING=gzip
--FILE--
<?php
ini_set('zlib.output_compression', 'Off');
$off = zlib_get_coding_type();
ini_set('zlib.output_compression', 'On');
$gzip = zlib_get_coding_type();
ini_set('zlib.output_compression', 'Off');
var_dump($off);
var_dump($gzip);
?>
--EXPECT--
bool(false)
string(4) "gzip"

Run this test again, and it finally passes! 🎊🥳🎉

This leaves you with writing one test case for the Accept-Encoding header being set to deflate and another test case for the Accept-Encoding header being set to an invalid string (basically something that is not gzip or deflate). But this is just diligence from this point on and if you like to cheat you can find the tests in the ext/zlib/tests/ directory named zlib_get_coding_type_basic.phpt, zlib_get_coding_type_br.phpt, zlib_get_coding_type_deflate.phpt and zlib_get_coding_type_gzip.phpt.

Collect Some Evidence!

Now that you have tests for all four cases you could hope you covered that function with tests or (a way better option if you ask me), you could continue and create a code coverage report! For this to work you need to install the lcov tool via your Linux package manager (sudo dnf install lcov for Fedora or sudo apt-get install lcov for Debian). As PHP was built with gcov disabled, we need to run configure again, with enabled gcov and recompile.

$ make clean
$ CCACHE_DISABLE=1 ./configure --with-zlib --enable-gcov
$ make -j `nproc`

This compiles the PHP binary with enabled code coverage generation. Lcov will be used to create an HTML code coverage report afterwards. To create your code coverage report, you need to rerun the tests.

$ make test TESTS=test/zlib_get_coding_type.phpt

While running the tests, gcov generates coverage files with a gcda file ending for every source code file. To generate the HTML code coverage report, run the following:

$ make lcov

You may find the resulting HTML code coverage report in lcov_html/index.html.

It looks like we covered all four cases.

What’s in It for You?

With every test you write PHP becomes more stable and reliable, as functions may not change behavior suddenly between releases.

Writing tests for PHP gives you a deeper understanding of how your favorite language works internally. In fact, by reading up to this point you learned that HTTP request headers are immutable to userspace, that there is such a thing as userspace, and that there are handler functions called and check what you are doing when you want to change ini settings at runtime.

Also knowing the PHPT file format gives you other benefits as well: PHPUnit not only supports the PHPT file format and can execute those tests, part of PHPUnit is tested with PHPT test cases. This led me to become a contributor to PHPUnit: Writig tests for PHPUnit in the PHPT file format.

Shout Out!

I would like to thank everyone involved in the PHP TestFest, but especially Ben Ramsey, Sammy Kaye Powers and everyone else involved in the organization of the 2017 edition. This was what made me write tests for PHP and made me not only a PHP but also a PHPUnit contributor. In the long run, this brought me my first ElePHPant 🐘!

Closing notes

I would like to thank my proof readers Kara Ferguson and Pim Elsfhof. It was a pleasure working with you!

"Wooble" is the property of Sammy Kaye Powers and is used with permission.

The German version is available through the PHP Magazin

Interested in seeing the talk to this blog post? Nomad PHP has you covered.

Learn how YOU can build a Serverless GraphQL API on top of a Microservice architecture with PHP

Florian Engelhardt — Mon, 06 May 2019 12:56:40 +0000

A few weeks earlier I read a very interesting article from Chris Noring about “Learn how YOU can build a Serverless GraphQL API on top of a Microservice architecture, part 1” which he entirely build with Node.js and Express and I thought to myself: you could do this in PHP and ReactPHP!

Disclaimer: This is not a JavaScript vs. PHP rant!

The big picture

As Chris Noring did in his original article, I created two microservices exposing a REST-API, one serving products and another one serving reviews to said products. Those two Microservices will be only available internal and not exposed to the internet, but nevertheless there should be customer facing API. For this purpose I will use GraphQL as a high level API combining the results of the two Services.

As you can imagine, there is nothing original in my article, I just did the same but with PHP and ReactPHP. You can find the source code on GitHub, so I will limit myself to the highlights.

Highlights

Services

The two services are pretty straight forward. Just a bit of ReactPHP-Code asynchronous handling incoming HTTP-Requests. For the sake of the example both are returning a static JSON-String. In a more in depth example you could connect to a database and look up the data.

GraphQL

When using GraphQL in PHP there are a couple of libraries, but only webonyx/graphql-php seems mature enough, also it has an asynchronous interface supporting ReactPHP. So this was the perfect match for this case.

When using the async-Interface, you need to call GraphQL::promiseToExecute instead of GraphQL::executeQuery and give it the promise adapter as a first parameter, which in my case was an instance of GraphQL\Executor\Promise\Adapter\ReactPromiseAdapter

When initialized and used this way the method call to promiseToExecute will return a ReactPHP Promise, on which you can register your then-Handler. Also your resolve-Functions can now return a ReactPHP Promise, which means you can write async code everywhere.

What you get is a pure asynchronous GraphQL-API written in PHP.

Find the source code at GitHub

realFlowControl / php-graphql-microservice

Simple example implementation of a GraphQL API on top of a microservice architecture

GraphQL API on top of Microservices

This is just a port from Node.js/Express to PHP/ReactPHP for Chris Norings article on building a Serverless GraphQL API on top of a Microservice architecture.

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

Prerequisites

You need to have docker and docker-compose up and running on your local machine.

Installing

git clone https://github.com/flow-control/php-graphql-microservice.git
cd php-graphql-microservice
make
docker-compose up -d
curl -X POST \
       -H "Content-Type: application/json" \
       --data '{ "query": "{ product (id:1) { id name } }" }' \
       localhost:8000

Build with

License

MIT, see LICENSE file.

View on GitHub

Happy coding!

Why using REPLACE INTO might be a bad idea

Florian Engelhardt — Mon, 11 Sep 2017 06:19:27 +0000

TTFB over all application servers (dashed line is previous period)

In our office we have a TV with a performance dashboard, so everyone can see possible problems in the system at any time. Amongst other things we show the TTFB on this dashboard. Those two spikes are not what you want to see, when you are just about to leave the office for lunch.

What happened?

We started looking around what’s going on and found out the following in that order:

the number of workers on the app-server rose dramatically
but the CPU consumption did not
the number of connections to the database reached the upper limit
but the CPU consumption did not rise (in fact it decreased)
and finally we start to see the following error messages in our monitoring tools:

Deadlock found when trying to get lock; try restarting transaction

by that time the system was not responding anymore
we checked the processlist from the database to see what it is doing
and found loads of REPLACE INTO statements and several big DELETE statements (running since minutes) on the same table

What is so special to that table?

It’s a cache lookup table. Everytime a page gets stored in our frontend cache, we do store some meta information about that cache entry in that database table, mostly dependencies to object data. And when that object data changes (for example the article stock), we fetch all cache entries depending on that object, flush them from the frontend cache and remove the information about it from the lookup table via a simple DELETE statement.

New entries in that lookup table are not created via INSERT, but via a REPLACE INTO statement. This is because it could happen, that two requests hit the same page, both are a cache miss and both try to create an entry in that lookup table. Easy solution …

The InnoDB gap lock

While digging deeper into that problem with our deadlocks, i found an article from Jervin Real from Percona about the InnoDB gap lock where he states the following:

For a non-INSERT write operation where the WHERE clause does not match any row, I expected there should’ve been no locks to be held by the transaction, but I was wrong

Until the moment i read that, i expected the same. But when you read that article from Jervin and the aging bug report in the MySQL bugzilla about the gap lock, you will understand that this makes perfect sense and you start to think about a REPLACE INTO statement being the right thing in this situation.

How we solved that problem?

We just replaced that REPLACE INTO statement with a INSERT ON DUPLICATE KEY UPDATE statement, which does exactly this. If the primary key is found, it updates that record, if not, the record gets inserted. This solved the problems with the deadlocks in that case.

You think that REPLACE INTO works exactly that way? Keep on reading.

The lesser known problem with `REPLACE INTO`

Do you know what REPLACE INTO really does? Let’s see:

CREATE TABLE a (
 id INT(1) UNSIGNED NOT NULL PRIMARY KEY,
 data VARCHAR(255) NULL
);

CREATE TABLE b (
 id INT(1) UNSIGNED NOT NULL PRIMARY KEY,
 fk INT(1) UNSIGNED NOT NULL,
 data VARCHAR(255) NULL,
 FOREIGN KEY (fk)
 REFERENCES a(id)
 ON DELETE CASCADE
);

INSERT INTO a VALUES (1, 'test');
INSERT INTO b VALUES (1, 1, 'test');

REPLACE INTO a VALUES (1, 'data');

What do you think will the result of a SELECT * FROM b; look like?

Exactly, the table is empty. This is because the REPLACE INTO first deletes the found record (which triggers the delete cascade) and then creates a new one. This might be expected behavior, depending on the situation, but to my understanding this would normally not be the intended behavior and at least this was new to me.

Delayed requeuing with RabbitMQ

Florian Engelhardt — Wed, 12 Apr 2017 13:33:24 +0000

Original at https://www.flickr.com/photos/criminalintent/3226840700/ (CC BY-SA 2.0)

The problem

Your consumer can not handle the received message and you need to reject it. If you reject the message with requeue being true, it will be instantly redelivered to your consumer, resulting in very high workload, as your consumer will reject that message. Things have come full circle 😋

Basicaly you want to reject the message and requeue it, but with a delay.

The solution

Well, you could not reject the message, but you could deliver it to another queue from your consumer and write a cronjob moving this messages back to the working queue. But there should be a better solution, and it should be possible to implement without any changes to your consumer or your message broker.

The real solution

Dead-Letter-Exchange (DLX) to the rescue!

For every queue you can specify a DLX, this is where messages that got rejected or deleted will be delivered to. In combination with a TTL you can automate delivery between queues with a delay.

To make this work, setup your exchanges and queues like so:

create an exchange named stuff and direct bind to the queue stuff
create an exchange named stuff-retry and direct bind to the queue stuff-retry
create a work queue named stuff with the x-dead-letter-exchange header set to stuff-retry
create a work queue named stuff-retry with the x-dead-letter-exchange header set to stuff and the x-message-ttl header to 300000

That’s it. If your consumer listening on stuff now rejects a message without requeue, it will be delivered to the stuff-retry-exchange which passes the message on to the stuff-retry-queue. After five minutes in that queue, the messages dies and will be redelivered to the stuff-exchange, which passes the message on to the stuff-queue so your consumer can process that message.

Block SSH brute-force attacks

Florian Engelhardt — Fri, 19 Aug 2016 09:53:32 +0000

… or at least slow them down (using iptables on Linux)

When having a server with the port 22 open to the internet, you will find a sheer endless number of login tries from various sources. 99% of these are just brute-force or dictionary attacks.

As I do not want my log files to boil over with all these failed logins, I searched for a solution to block them out or at least slow them down.

Here is what I did (thanks @baptistecs for the missing ACCEPT and adding IPv6)

# cleanup
iptables -F
iptables -X SSH_CHECK

ip6tables -F
ip6tables -X SSH_CHECK

# set rules
iptables -N SSH_CHECK
iptables -A SSH_CHECK -m recent --set --name SSH
iptables -A SSH_CHECK -m recent --update --seconds 60 --hitcount 2 --name SSH -j DROP
iptables -A SSH_CHECK -m recent --update --seconds 3600 --hitcount 10 --name SSH -j DROP
iptables -A SSH_CHECK -p tcp --dport 22 -j ACCEPT # accept packet if not previously dropped

ip6tables -N SSH_CHECK
ip6tables -A SSH_CHECK -m recent --set --name SSH
ip6tables -A SSH_CHECK -m recent --update --seconds 60 --hitcount 2 --name SSH -j DROP
ip6tables -A SSH_CHECK -m recent --update --seconds 3600 --hitcount 10 --name SSH -j DROP
ip6tables -A SSH_CHECK -p tcp --dport 22 -j ACCEPT # accept packet if not previously dropped

iptables -A INPUT -p tcp -s 127.0.0.1 -j ACCEPT # whitelist your IP (replace 127.0.0.1)
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -j SSH_CHECK # jump from INPUT to SSH_CHECK

ip6tables -A INPUT -p tcp -s ::1/128 -j ACCEPT # whitelist your IP (replace ::1/128)
ip6tables -A INPUT -p tcp --dport 22 -m state --state NEW -j SSH_CHECK # jump from INPUT to SSH_CHECK

I am creating a new chain called SSH_CHECK to which i pass on every TCP packet received on port 22 that is in state new (aka syn packet). The source IP address is logged and if the same address occurs two or more times in the last minute or ten or more times in the last hour that packet is dropped.

I also added a rule to bypass the SSH_CHECK chain if the source IP address equals 127.0.0.1 / ::1/128 (replace that with your static IP or just ditch that line).

To be honest, I am not blocking brute-force attacks, but the attacker (or attacking script) can only try one password per minute or nine passwords per hour.

And at last: it is generally a good advice to turn of password authentication in your ssh server as well as root login. Add the following to your /etc/ssh/sshd_config file or change it to read as follows:

PasswordAuthentication no
PermitRootLogin no
PubkeyAuthentication yes

DEV Community: Florian Engelhardt

To double quote or not, that's the question!

What is the fuzz all about?

Case closed?

The lexer

The parser

The compiler

Hypothesis

What happens at runtime?

New Hypothesis

Get to the point: should I concat or interpolate?

TLDR

Final advice

What about sprintf()?

Processing One Billion Rows in PHP!

A first naive approach

fgets() instead of fgetcsv()

Using a reference where possible

Only one comparison

Adding type casts

What about JIT?

What more?

Can we make it even faster?

Is this the end?

What did I learn on the way?

The END?

There is MOAR!! performance to gain

Remove the isset()

Don't check on the fgets() return value

fgets() vs. stream_get_line()

The END again?

Revisiting GitLab as a PHP Developer

TLDR

Unit test reports

Test code coverage

Test code coverage visualization

Code style violations

Growing the PHP Core—One Test at a Time

Prepare Your machine!

What Do Tests Look Like?

The TEST Section

The SKIPIF Section

The FILE Section

The EXPECT Section

The EXPECTF Section

The XFAIL Section

The CLEAN Section

What else?

What Can You Test?

Time To Write That Test!

Collect Some Evidence!

What’s in It for You?

Shout Out!

Closing notes

Learn how YOU can build a Serverless GraphQL API on top of a Microservice architecture with PHP

The big picture

Highlights

Services

GraphQL

realFlowControl / php-graphql-microservice

Simple example implementation of a GraphQL API on top of a microservice architecture

GraphQL API on top of Microservices

Getting Started

Prerequisites

Installing

Build with

License

Why using REPLACE INTO might be a bad idea

What happened?

What is so special to that table?

The InnoDB gap lock

How we solved that problem?

The lesser known problem with REPLACE INTO

Delayed requeuing with RabbitMQ

The problem

The solution

The real solution

Block SSH brute-force attacks

What about `sprintf()`?

`fgets()` instead of `fgetcsv()`

Remove the `isset()`

Don't check on the `fgets()` return value

`fgets()` vs. `stream_get_line()`

The lesser known problem with `REPLACE INTO`