There are some fairly new players in the PHP runtimes team. It’s interesting to understand how good they are in comparison to well-known and widely used.
The main questions I wanted to get answers are:
- How many requests per second the runtime can process?
- What’s the average response time each runtime can provide?
To have fair competition, all runtimes are given the same resources. Almost. If the runtime consists of only 1 container(ex. Nginx Unit), it is run with 1CPU and 1GB of RAM. If the runtime consists of 2 containers(ex. Nginx + PHP-FPM), each container is run with 1 CPU and 1GB of RAM.
The testing environment
- Each runtime is run in 1 or 2 Docker containers.
- Each container has 1 CPU and 1GB of RAM.
- The load testing tool is in the same Docker network
Testing application.
Let’s try something more complex than single-file Hello-World. So, we have:
- Symfony 7
- 1 controller
- 1 view
- no database or any other external service
Symfony provides the runtime component. According to the official doc it “decouples the bootstrapping logic from any global state to make sure the application can run with runtimes like”. It means, that you can develop the application using any runtime you like, but in production run the most performant. So, we can run the application not only with traditional servers, but with libraries like Swoole, AMPHP, and ReactPHP. Looking ahead, the last two were excluded from the competition.
Load testing tool
K6 was used to run load tests. It was run 3 times for each runtime, with 10, 100, and 1000 concurrent connections within 30 seconds.
Runtimes
- Apache(prefork mode) + mod_php.
- Apache(event mode) + PHP-FPM
- Nginx + PHP-FPM
- Nginx Unit application server.
- Roadrunner
- Nginx + Roadrunner (fcgi mode)
- FrankenPHP
- FrankenPHP (worker mode)
- Swoole
All runtimes are based on official docker images. Volumes are not used, the code of the application is copied to the image during the build. PHP 8.3 is everywhere, except Nginx Unit. At the beginning of 2024 the highest PHP version supported by Nginx Unit is 8.2.
Unfortunately, I didn’t find the up-to-date versions of ReactPHP and AMPHP runtimes compatible with Symfony 7. PHPPM both GitHub and Dockerhub look abandoned.
The following PHP and Symfony settings are applied:
- php.ini-production (comes with official docker containers) is used
- Opcache enabled
- JIT enabled
- preload is configured according to Symfony best practices
- Composer autoloader is optimized
- Symfony is run in production mode
- Service container is dumped to a single file
- Symfony cache is warmed up during container build
The application code, as well as container configs, can be found on GitHub.
Results
- 001_Apache+mod_php and 002_Apache + PHP-FPM showed almost the same results.
- 003_Nginx+PHP-FPM is very close to 001_Apache+mod_php and 002_Apache + PHP-FPM when the workload is low (concurrency 10 and 100 ).
- 003_Nginx+PHP-FPM was able to serve ~2 times more requests than Apache-based stacks when the concurrency is 1000
- 004_Nginx Unit. First of all — WOW!!! It is ~3 times faster than the traditional and most popular 003_Nginx+PHP-FPM.
- I don’t see a big difference between 005_Roadrunner and 006_Nginx+Roadrunner.
- Nevertheless, Roadrunner-based stacks are more than ~2 times faster than 003_Nginx+PHP-FPM.
- 007_FrankenPHP(non-worker mode) is pretty close to 003_Nginx+PHP-FPM from performance point of view
- 008_FrankenPHP(worker mode). Double WOW!!! I didn’t believe from the first attempt and ran load tests 3 or 4 times. So, when sending 1000 concurrent requests FrankenPHP(worker mode) is >10 times faster than Nginx+PHP-FPM. Also, faced an issue. Couldn’t start the container with php-ini.production.
- 009_swoole — also double WOW!. Almost the same speed as 008_FrankenPHP(wm). Extremely fast. But, keep in mind that your code should be adjusted to be run with Swoole. For Symfony we have a bundle, which extends symfony/runtime component.
Personal opinion.
- FrankenPHP — amazing job, the first candidate to become standard de facto in PHP world.
- Swoole — I want it to be out of the box (but disabled) PHP extension.
- Nginx Unit — my personal choice. Despite it doesn’t support HTTP2 and many other features traditional Nginx provides, it is super easy to configure, light, and fast.
What’s next?
It is worth trying the same tests with an application connected to the database.
Try ReactPHP, AMPHP, and PHPPM-based runtimes. But before that, the appropriate packages need to be updated to work with Symfony 7.
Run containers with more resources, let’s say 2–4 CPU and 4–8GB of RAM. It would be interesting to see how well the runtimes can scale.
Top comments (5)
Hey, I would like to point on some mistakes in your RoadRunner benchmarks.
Interesting comparison, thanks!
As always, it brings a lot of questions as well, which could lead down the rabbit hole of we-need-more-details-and-more-tests-too ;-)
My notes:
is it fair to compare configurations with 2 containers with configs running a single one? The former ones will most likely score better on the tests with 1K connections, as they have more total memory available, but score worse with 10 connections, as they have the overhead of passing through the docker network stack... Imho it would be fairer to at least give each of the containers in the 2-containers settings half the memory
while I understand wanting to use the official docker containers for each runtime, I wonder if that is the best choice, or the one that would be most used for production workloads. Fe. I tend to run nginx and php-fpm within a single container - which means that they can talk over a unix socket instead of the tcp stack. This should give an improvement in the figures
it is not immediately clear from your description which, if any, of the runtimes need a restart/reload command whenever a php file gets updated, whereas others will pick up immediately and automatically any change. Maybe this is a moot point given the Symfony setup for the app in use (does it mandates a full app rebuild and redeploy on changes to each and every php file?), but it is an important distinction to make in general
the opcache php configuration can make a great deal of difference, as well as the realpath_cache one. Where those checked to be optimal for the app at hand (eg. allow to cache all php files), and did they avoid hitting the disk often to check for file staleness (again, this is a choice which different users might have different opinions about)?
what about the number of workers processes and the number of requests each other will serve before "suiciding": was this set up to be the same on every server?
Hey, good article, but I'd like to give my 5 cents to the discussion:
1) were all the platforms tested on 1 CPU core with one worker and 1 thread (for FrankenPHP)? I think a reasonable comparison is only with such kind of a setup
2) for Swoole, maybe instead of SF runtime, this bundle could be tested: github.com/symfony-swoole/swoole-b... You can also enable swoole coroutines there and see the impact yourself, it would blow your mind a little I guess :)
Thank you for sharing! Very insightful
Please, check the PRs in the github repo to have a fair bench !!
Thank you !!