DEV Community

Cover image for Not a DDoS but an OPcache bug: 600 MB PHP-FPM workers on a server with 5,500 WordPress sites
Oleg Usoltsev
Oleg Usoltsev

Posted on Originally published at habr.com

Not a DDoS but an OPcache bug: 600 MB PHP-FPM workers on a server with 5,500 WordPress sites

The client wrote: the sites are down, looks like a DDoS. The server runs about 5,500 WordPress sites on nginx, PHP-FPM 8.4 and MariaDB, with 125 GB of RAM on Debian 13. I found a scanner, blocked it in nginx, and the sites came back. Two days later the client wrote again: now everything falls over every five minutes.

The cause was not outside. Every new PHP-FPM worker weighed about 600 MB from its very first request, and the OOM killer took turns shooting the database and nginx. The workers were inflated by a bug in OPcache that fires on every cache reset under load. The resets came from Yoast SEO during an update. Not the regular update across all sites, but a catch-up run on 467 sites where the scheduled one had failed. In the end, the order of two commands decided everything.

Below is how I found it, step by step, and what I changed. The command output is real; I removed domains and addresses.

The server

All sites are served by one PHP-FPM pool:

pm.max_children = 200
pm.start_servers = 20
pm.min_spare_servers = 10
pm.max_spare_servers = 30
pm.max_requests = 500
Enter fullscreen mode Exit fullscreen mode

And one 1 GB OPcache:

opcache.enable=1
opcache.memory_consumption=1024
opcache.max_accelerated_files=200000
opcache.validate_timestamps=1
opcache.revalidate_freq=0
Enter fullscreen mode Exit fullscreen mode

There is one FPM master, so the OPcache shared memory is also one for all 5,500 sites. Sizes below are the way ps and /proc count them: 1 MB is 1024 KB.

First call: a scanner

Load average 205-235 on 32 cores, all 200 workers busy, 116 GB of memory used, swap full. About 34 requests per second from outside across 1,536 domains. More than half were looking for .env, .git and other service files, from hundreds of cloud addresses.

An ordinary scanner, they walk every public server. But on WordPress each such request turns into a full 404 page render in PHP, so the fastest fix is to stop them before PHP: a map in nginx and one line in the snippet every virtual host includes.

The map and the snippet line
map $uri $scanner_block {
    default 0;
    "~^/\.well-known/" 0;
    "~/\." 1;
    "~*(phpinfo|serverless\.ya?ml|docker-compose)" 1;
    "~*\.(sql|bak|swp|old|orig|env|ya?ml)$" 1;
}
Enter fullscreen mode Exit fullscreen mode
if ($scanner_block)          { return 404; }
Enter fullscreen mode Exit fullscreen mode

/.well-known/ is carved out so certificate issuance keeps working. I return 404 rather than 444: there is a load balancer and Cloudflare in front, and a dropped connection would turn into a 502 for a real visitor.

In the first minute with the filter, nginx itself answered 1,013 requests with 404, there were 293 normal responses and not a single 5xx. Some services had been killed by OOM along the way and were brought back. It looked like the usual scanner story.

Second call: "every five minutes"

Two days later: DDoS again, now it goes down every five minutes. A different scanner, this one tries archives:

"GET /phpinfo.zip HTTP/2.0" status=404
"GET /cvs.zip HTTP/2.0" status=404
"GET /usermanager.zip HTTP/2.0" status=404
"GET /fortune.zip HTTP/2.0" status=499
Enter fullscreen mode Exit fullscreen mode

I could have added archives to the same map and moved on. But a couple of days later a third scanner would show up. I asked the client not to touch anything on the server, set up a script that logs the memory of nginx, PHP and the database every five minutes, and started with the kernel log.

Who gets killed

# journalctl -k --since 2026-10-01 | grep -c 'Killed process'
82
# journalctl -k --since 2026-10-01 | grep 'Killed process' | grep -o '([a-z()-]*)' | sort | uniq -c | sort -rn
     36 (mariadbd)
     22 (systemd)
     17 ((sd-pam))
      7 (nginx)
# journalctl -k --since 2026-10-01 -o short-iso | grep 'Killed process' | cut -c1-10 | uniq -c
     18 2026-10-12
     48 2026-10-13
     16 2026-10-15
Enter fullscreen mode Exit fullscreen mode

82 kills since the start of the month, all of them in the last four days. Before that the OOM killer was silent. None on the day after the filter, then again. This is what the morning of the second call looked like:

# journalctl -k --since 2026-10-15 --until '2026-10-15 13:00' | grep 'Killed process'
Oct 15 04:21:01 srv kernel: Out of memory: Killed process 1597116 (mariadbd) total-vm:48713652kB, anon-rss:23905292kB, ...
Oct 15 08:23:20 srv kernel: Out of memory: Killed process 2704383 (mariadbd) total-vm:44486476kB, anon-rss:23059620kB, ...
Oct 15 08:23:27 srv kernel: Out of memory: Killed process 3285965 (systemd) total-vm:22132kB, anon-rss:2048kB, ...
Oct 15 12:19:52 srv kernel: Out of memory: Killed process 3847971 (mariadbd) total-vm:43939700kB, anon-rss:13486656kB, ...
Oct 15 12:47:58 srv kernel: Out of memory: Killed process 1661507 (systemd) total-vm:22140kB, anon-rss:2048kB, ...
Oct 15 12:47:58 srv kernel: Out of memory: Killed process 1661509 ((sd-pam)) total-vm:22436kB, anon-rss:1032kB, ...
Oct 15 12:47:58 srv kernel: Out of memory: Killed process 430332 (mariadbd) total-vm:44793656kB, anon-rss:10777012kB, ...
Oct 15 12:48:04 srv kernel: Out of memory: Killed process 3776728 (nginx) total-vm:3247688kB, anon-rss:2518036kB, ...
Enter fullscreen mode Exit fullscreen mode

The database is the biggest process, so it gets killed most often, 36 times out of 82. MariaDB's unit has Restart=on-abnormal and RestartSec=5s: systemd brings it back, it grows again, and the cycle repeats. Every database kill means a minute or two of errors on all sites while it starts. That is the "goes down every five minutes" from the client's message.

systemd and (sd-pam) in the list are not PID 1 but a user session manager. systemd raises its kill priority:

# systemctl show user@1003.service -p OOMScoreAdjust
OOMScoreAdjust=100
Enter fullscreen mode Exit fullscreen mode

For such a process the kernel adds a tenth of total memory to its score, so a process of a few megabytes looks to the OOM killer like one of 12+ GB. That puts it in line right next to the database, even though it uses almost nothing.

nginx is more interesting. The kernel kills one of its processes, and systemd does the rest:

# journalctl -u nginx -S '2026-10-15 12:47' -U '2026-10-15 14:00'
Oct 15 12:48:04 srv systemd[1]: nginx.service: A process of this unit has been killed by the OOM killer.
Oct 15 12:48:09 srv systemd[1]: nginx.service: Stopping timed out. Terminating.
Oct 15 12:48:14 srv systemd[1]: nginx.service: State 'stop-sigterm' timed out. Killing.
Oct 15 12:48:14 srv systemd[1]: nginx.service: Killing process 1661457 (nginx) with signal SIGKILL.
Oct 15 12:48:19 srv systemd[1]: nginx.service: Processes still around after SIGKILL. Ignoring.
Oct 15 12:48:24 srv systemd[1]: nginx.service: Main process exited, code=killed, status=9/KILL
Oct 15 12:48:24 srv systemd[1]: nginx.service: Failed with result 'oom-kill'.
Oct 15 12:48:24 srv systemd[1]: nginx.service: Consumed 1h 20min 47.726s CPU time, 33.1G memory peak, 883.5M memory swap peak.
Enter fullscreen mode Exit fullscreen mode

The service has OOMPolicy=stop, which is the default: after any of its processes is killed, systemd stops the whole service. And Restart=no, so the sites stay down until someone starts nginx by hand.

It's not the bots

My first thought was the simplest one: the bots got meaner. proxy-audit.log here is a shared nginx request log for all sites (address, X-Forwarded-For, host, request, status), rotated daily. I counted archive requests per day in it, and upstream timeouts to PHP in the nginx error log:

# zgrep -cE '\.(zip|rar|7z|tar|gz|tgz|bz2) HTTP' /var/log/nginx/proxy-audit.log.[2-9].gz /var/log/nginx/proxy-audit.log.1 /var/log/nginx/proxy-audit.log
# zgrep -c 'upstream timed out' /var/log/nginx/error.log.[2-9].gz
Enter fullscreen mode Exit fullscreen mode
Days ago Archive requests upstream timed out
9 12,235 0
8 6,669 0
7 16,573 0
6 8,538 0
5 8,941 0
4 415 0
3, first outage 6,140 14
2, first call 6,442 8,325
1 8,850 not counted
today, until evening 14,892 not counted

Four days before the first outage there were 16,573 archive requests, more than on any outage day, and not a single timeout. Timeouts appeared on the day of the first outage.

And a back-of-the-envelope check. A clean 404 through FPM, as the measurement below shows, takes 0.29 seconds. At 34 requests per second, Little's law says about ten workers would cover it (34 × 0.29 ≈ 10). The first time, all 200 were busy, which means almost 6 seconds per request (200 / 34 ≈ 5.9). What made requests slow was not their complexity but the state of the server: memory gone, swap full, the database restarting every few minutes. The bot theory was wrong: nothing changed outside, something changed inside.

A 600 MB worker from the first request

# ps -o pid,etimes,rss,time -C php-fpm8.4 --sort=rss | head -8
    PID ELAPSED   RSS     TIME
3532685 1681700 11728 00:08:54
2655701       5 594932 00:00:00
2655702       5 603992 00:00:00
2655700       6 605512 00:00:00
2551941     305 648276 00:00:10
2556065     291 698096 00:00:10
2481721     519 704116 00:00:18
Enter fullscreen mode Exit fullscreen mode

The first line is the master, 11 MB. Then the workers: alive for 5 seconds, less than a second of CPU time, and they weigh almost 600 MB. The older ones are at 630-690 MB.

Maybe the request itself is heavy? I sent one request for a non-existent .zip, exactly like the scanner, straight into the FPM socket with cgi-fcgi. A small script fakes $_SERVER, loads WordPress and writes stats to a file at the end:

# SCRIPT_FILENAME=/tmp/wp404-probe.php REQUEST_METHOD=GET PROBE_HOST=example.com PROBE_URI=/probe-20261015-xq7.zip timeout 110 cgi-fcgi -bind -connect /run/php/php8.4-fpm.sock
# cat /tmp/wp404-probe.out
{
    "time_s": 0.29,
    "peak_mb_real": 36,
    "peak_mb": 34,
    "queries": 133,
    "is_404": true,
    "included_files": 1464,
    ...
}
Enter fullscreen mode Exit fullscreen mode

The full wp404-probe.php
<?php
$host = getenv('PROBE_HOST') ?: 'example.com';
$uri  = getenv('PROBE_URI') ?: '/probe-20261015-xq7.zip';
$root = "/var/www/$host";
foreach (['HTTP_HOST' => $host, 'SERVER_NAME' => $host, 'REQUEST_URI' => $uri, 'REQUEST_METHOD' => 'GET',
          'HTTPS' => 'on', 'SERVER_PORT' => '443', 'SCRIPT_NAME' => '/index.php', 'PHP_SELF' => '/index.php',
          'SCRIPT_FILENAME' => "$root/index.php", 'DOCUMENT_ROOT' => $root, 'QUERY_STRING' => '',
          'HTTP_USER_AGENT' => 'wp404-probe', 'REMOTE_ADDR' => '127.0.0.1'] as $k => $v) {
    $_SERVER[$k] = $v;
}
unset($_SERVER['PATH_INFO'], $_SERVER['ORIG_PATH_INFO']);
$t0 = microtime(true);
register_shutdown_function(function () use ($t0, $host, $uri) {
    global $wpdb;
    $st = function_exists('opcache_get_status') ? opcache_get_status(false) : null;
    $err = error_get_last();
    $out = [
        'host' => $host, 'uri' => $uri,
        'time_s' => round(microtime(true) - $t0, 2),
        'peak_mb_real' => round(memory_get_peak_usage(true) / 1048576, 1),
        'peak_mb' => round(memory_get_peak_usage(false) / 1048576, 1),
        'queries' => isset($wpdb) ? $wpdb->num_queries : null,
        'is_404' => function_exists('is_404') ? is_404() : null,
        'included_files' => count(get_included_files()),
        'last_error' => $err ? substr($err['message'], 0, 200) . ' @' . $err['file'] . ':' . $err['line'] : null,
        'opcache' => $st ? [
            'used_mb' => round($st['memory_usage']['used_memory'] / 1048576),
            'free_mb' => round($st['memory_usage']['free_memory'] / 1048576),
            'wasted_mb' => round($st['memory_usage']['wasted_memory'] / 1048576),
            'wasted_pct' => round($st['memory_usage']['current_wasted_percentage'], 1),
            'interned_used_mb' => round($st['interned_strings_usage']['used_memory'] / 1048576),
            'interned_free_mb' => round($st['interned_strings_usage']['free_memory'] / 1048576),
            'cache_full' => $st['cache_full'],
            'restart_pending' => $st['restart_pending'],
            'scripts' => $st['opcache_statistics']['num_cached_scripts'],
            'keys' => $st['opcache_statistics']['num_cached_keys'],
            'max_keys' => $st['opcache_statistics']['max_cached_keys'],
            'hits' => $st['opcache_statistics']['hits'],
            'misses' => $st['opcache_statistics']['misses'],
            'blacklist_misses' => $st['opcache_statistics']['blacklist_misses'],
            'hit_rate' => round($st['opcache_statistics']['opcache_hit_rate'], 2),
            'oom_restarts' => $st['opcache_statistics']['oom_restarts'],
            'hash_restarts' => $st['opcache_statistics']['hash_restarts'],
            'manual_restarts' => $st['opcache_statistics']['manual_restarts'],
            'start_time' => date('c', $st['opcache_statistics']['start_time']),
            'last_restart' => date('c', $st['opcache_statistics']['last_restart_time']),
        ] : null,
    ];
    file_put_contents('/tmp/wp404-probe.out', json_encode($out, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES) . "\n", FILE_APPEND);
});
ob_start();
chdir($root);
require "$root/index.php";
Enter fullscreen mode Exit fullscreen mode

The script sat in /tmp only for the measurement, then I deleted it.

0.29 seconds and 36 MB at peak. PHP does not know about its 600 MB, memory_get_peak_usage() does not see them. So the memory lives somewhere below PHP.

Where the 500 MB are

# grep -E '^(VmRSS|RssAnon|RssFile|RssShmem)' /proc/2481721/status
VmRSS:    648724 kB
RssAnon:      533656 kB
RssFile:       30196 kB
RssShmem:      84872 kB
Enter fullscreen mode Exit fullscreen mode

RssShmem is the OPcache shared memory, all workers share it. But the 521 MB of RssAnon belong to each worker alone. I looked in /proc/2481721/smaps for regions larger than 10 MB. There are just two: /dev/zero (deleted) at 84,868 KB, which is that OPcache shared memory, and a nameless 7f6cdd686000-7f6cfc6a7000 at 508,032 KB. Its smaps entry, without zero and housekeeping fields:

7f6cdd686000-7f6cfc6a7000 rw-p 00000000 00:00 0
Size:             508036 kB
Rss:              508032 kB
Private_Dirty:    508032 kB
Anonymous:        508032 kB
AnonHugePages:    505856 kB
Swap:                  0 kB
SwapPss:               0 kB
THPeligible:           1
VmFlags: rd wr mr mw me ac sd
Enter fullscreen mode Exit fullscreen mode

A 496 MB anonymous chunk, fully populated: Rss equals Size. How many processes hold a region at this address:

# grep -l '7f6cdd686000-7f6cfc6a7000' /proc/[0-9]*/maps | wc -l
30
# ps -C php-fpm8.4 --no-headers | wc -l
31
Enter fullscreen mode Exit fullscreen mode

All 30 workers; the 31st process is the master. The master has 5 MB of RssAnon:

# grep -E '^(VmRSS|RssAnon|VmData|VmSize)' /proc/3532685/status
VmSize:  1261320 kB
VmRSS:     11728 kB
RssAnon:        5344 kB
VmData:    12440 kB
Enter fullscreen mode Exit fullscreen mode

So a worker allocates this chunk after fork, on its very first request, and fills it right away. This is not a leak that builds up over the life of a process. The worker pays a tax at birth.

The php-src bug, #23859

I searched for the symptom and found a php-src issue: #23859, "Repeated opcache_reset() under concurrency permanently grows ZCSG(map_ptr_last), inflating every newly spawned php-fpm worker". It had been opened three weeks before my investigation, status Needs Triage, no patch. The author's workers weighed about 960 MB at birth, the whole FPM was at 52 GB, and a graceful reload brought it down to 2.4 GB. They reproduced the growth on 8.1, 8.3, 8.4 and 8.5, including Debian 13's 8.4, and honestly call the code path a hypothesis. The first commenter laid out the mechanism with line references. I checked it against the 8.4.26 source running on this server. Step by step:

  1. Every PHP process has a map_ptr table: one slot for each function, method and class from the OPcache cache. The process that puts a script into the cache takes new slots from its own counter CG(map_ptr_last) and writes it into the shared counter ZCSG(map_ptr_last) (zend_persist.c:1457).
  2. A new worker, loading its first script from the cache, stretches its table up to the shared counter (zend_accelerator_util_funcs.c:388). At the start of every request the whole table is zeroed (zend.c:1321), so all of it is resident at once.
  3. A cache restart zeroes the shared counter, but only in the process that performs the restart (ZendAccelerator.c:2714). The other workers notice that the cache was restarted and reset only their path and regex caches (2748-2756). Their own counter stays at the old value.
  4. The first of them to put a script into the cache allocates slots on top of that old value and writes it into the shared counter. The counter never goes down, and every restart under load adds another generation to it. This lasts until FPM itself is restarted.

Any restart counts here: opcache_reset(), the cache running out of memory, the hash table overflowing. All three go through the same block of code (2695-2730). While a restart waits for all processes to let go of the cache, OPcache is disabled for everyone (3473-3476), and new resets are not scheduled in the meantime, the source literally says "don't schedule twice" (3459). That is why there are noticeably more opcache_reset() calls than restarts in the stats.

The bug leaves a recognizable fingerprint. The table grows in steps of 4096 pointers, that is 32 KB (zend.c:2063), and 508,036 KB is exactly 15,876 steps of 32 KB plus 4 KB. If you see a nameless chunk of that kind of size, identical in every worker and missing in the master, it is almost certainly this.

In the author's synthetic test, 1,800 resets give about 200 MB per worker. Here a couple of hundred restarts were enough. Reading the code, a generation gets longer the more scripts manage to get back into the cache before the next restart, and this server's cache holds 45 thousand scripts. That is my reading of the code; I did not measure it.

There is one more cost. Since the table is zeroed at the start of every request, every request does a memset over 496 MB. My estimate is tens of milliseconds of CPU per request; I did not measure that either.

Who resets the cache

OPcache stats have to come from FPM itself. php -r in a console is useless: the CLI has its own OPcache, usually disabled. The same probe writes them:

"opcache": {
    "used_mb": 979,
    "free_mb": 0,
    "wasted_pct": 4.4,
    "cache_full": true,
    "scripts": 45037,
    "hit_rate": 34.72,
    "oom_restarts": 2,
    "hash_restarts": 0,
    "manual_restarts": 205,
    "start_time": "2026-10-09T07:39:32+00:00",
    "last_restart": "2026-10-13T12:15:07+00:00"
}
Enter fullscreen mode Exit fullscreen mode

The cache is completely full, two thirds of lookups miss, and those scripts get compiled again on every request. manual_restarts: 205 are restarts by opcache_reset(), oom_restarts: 2 are two more from the cache itself running out of memory, and they take the same path. All of it counts from start_time, the last FPM restart.

I looked for who calls opcache_reset() with a plain grep over one site:

# grep -rn opcache_reset wp-content/plugins wp-content/themes
wp-content/plugins/wordpress-seo/wp-seo-main.php:344:       if ( $should_invalidate_opcache && function_exists( 'opcache_reset' ) ) {
wp-content/plugins/wordpress-seo/wp-seo-main.php:346:           @opcache_reset();
Enter fullscreen mode Exit fullscreen mode

I went over one copy of every plugin present on the server, all three themes and the mu-plugins. Besides Yoast, only Google Site Kit turned up: it resets the cache after any update through the upgrader_process_complete hook, but it is installed on one site out of 5,500. Fun detail: right next to the call they have a comment saying "opcache_reset is prohibited on the WordPress VIP platform due to memory corruption".

Yoast's code (wp-seo-main.php, lines 329-346), shortened:

if ( version_compare( WPSEO_Options::get( 'version', 1, [ 'wpseo' ] ), WPSEO_VERSION, '<' ) ) {
    // Invalidate the opcache in 50% of the cases, randomly staggered based on the site URL.
    $random_seed               = hexdec( substr( hash( 'sha256', site_url() ), 0, 8 ) );
    $should_invalidate_opcache = ( $random_seed % 2 ) !== 0;
    $should_invalidate_opcache = (bool) apply_filters( 'Yoast\WP\SEO\should_invalidate_opcache', $should_invalidate_opcache );
    if ( $should_invalidate_opcache && function_exists( 'opcache_reset' ) ) {
        @opcache_reset();
    }
    new WPSEO_Upgrade();
Enter fullscreen mode Exit fullscreen mode

After an update, the first WordPress load sees that the stored version is older than the code, and on half of the sites it resets OPcache. Which half is decided by a hash of the site URL, so it is always the same half. For a single site on its own server that is reasonable: you do not want to live with a mix of old and new files in the cache. Note the words "the first WordPress load", not "the first request". That is the key to the rest.

Why it didn't fall over before

# journalctl -u php8.4-fpm --since 2026-10-01 | grep Reloaded
Oct 01 05:23:08 srv systemd[1]: Reloaded php8.4-fpm.service - The PHP 8.4 FastCGI
Oct 01 05:25:17 srv systemd[1]: Reloaded php8.4-fpm.service - The PHP 8.4 FastCGI
Oct 02 14:12:44 srv systemd[1]: Reloaded php8.4-fpm.service - The PHP 8.4 FastCGI
Oct 06 07:01:34 srv systemd[1]: Reloaded php8.4-fpm.service - The PHP 8.4 FastCGI
Oct 08 05:44:48 srv systemd[1]: Reloaded php8.4-fpm.service - The PHP 8.4 FastCGI
Oct 09 07:12:06 srv systemd[1]: Reloaded php8.4-fpm.service - The PHP 8.4 FastCGI
[...]
Oct 09 07:39:31 srv systemd[1]: Reloaded php8.4-fpm.service - The PHP 8.4 FastCGI
Oct 15 18:14:41 srv systemd[1]: Reloaded php8.4-fpm.service - The PHP 8.4 FastCGI
Enter fullscreen mode Exit fullscreen mode

PHP-FPM used to be reloaded every one to four days, and each reload reset the table: the OPcache shared memory is created fresh, the counter starts from zero, and nobody ever sees the inflation. The last reload before the outages was three days before the first one. The last line in the output is mine, more on it below.

What reset the cache during those three days is not obvious from the logs: OPcache resets are not logged by name anywhere. I had to piece it together from update logs, the sudo journal and OOM killer reports.

The sites here are updated by a cron script: every five minutes it checks for new versions and installs them on all sites. Its log for the previous Yoast update, two weeks before the outages, and for the next one, the day after the first outage:

# grep -E 'wordpress-seo|Updated|FAILED' 2026-09-29_12-40-01/main.log 2026-10-13_11-30-01/main.log
2026-09-29_12-40-01/main.log:[2026-09-29 12:40:03]   wordpress-seo: 28.5 -> 28.6
2026-09-29_12-40-01/main.log:[2026-09-29 12:40:03]   [DOWNLOAD] wordpress-seo 28.6
2026-09-29_12-40-01/main.log:[2026-09-29 12:40:04]   [OK] wordpress-seo 28.6
2026-09-29_12-40-01/main.log:[2026-09-29 13:22:31]   Updated:         5058
2026-09-29_12-40-01/main.log:[2026-09-29 13:22:31] FAILED:            469
2026-10-13_11-30-01/main.log:[2026-10-13 11:30:03]   wordpress-seo: 28.6 -> 28.7
2026-10-13_11-30-01/main.log:[2026-10-13 11:30:03]   [DOWNLOAD] wordpress-seo 28.7
2026-10-13_11-30-01/main.log:[2026-10-13 11:30:04]   [OK] wordpress-seo 28.7
2026-10-13_11-30-01/main.log:[2026-10-13 12:16:41]   Updated:         5478
2026-10-13_11-30-01/main.log:[2026-10-13 12:16:41] FAILED:            7
Enter fullscreen mode Exit fullscreen mode

On 469 sites the update to 28.6 failed, and 467 of them have [PLUGIN:wordpress-seo] ERROR: update failed in the site log. Cron did not retry them: the next run with Yoast in the logs is two weeks later, already for 28.7.

The lagging sites were caught up by a manual run on the morning of the first outage: wp plugin update --all across all sites and a couple of wp-config.php edits. The order of WP-CLI commands on one of the lagging sites, from the sudo journal:

# journalctl _COMM=sudo -S '2026-10-12 06:25:30' -U '2026-10-12 06:26' | grep 'PWD=/var/www/<site> ' | grep -o 'COMMAND=/usr/local/bin/wp.*'
COMMAND=/usr/local/bin/wp option update tfa_administrator 1
COMMAND=/usr/local/bin/wp config set WP_DEBUG false --raw
COMMAND=/usr/local/bin/wp config set WP_DEBUG_LOG false
COMMAND=/usr/local/bin/wp plugin update --all
Enter fullscreen mode Exit fullscreen mode

wp plugin update --all is the last command, and after it WP-CLI never loads the site again. So Yoast's upgrade code, together with opcache_reset(), was run by the site's first visitor, which means by PHP-FPM, under live traffic. That the lagging sites got 28.6 exactly then shows up in the site logs of the next run:

# grep -l '28.5 -> 28.7' 2026-10-13_11-30-01/sites/*.log | wc -l
2
# grep -l '28.6 -> 28.7' 2026-10-13_11-30-01/sites/*.log | wc -l
5483
Enter fullscreen mode Exit fullscreen mode

There are no other Yoast updates between these runs in the sudo journal; I checked a sample of ten lagging sites. So about 467 sites were updated that morning, and roughly half of them, per Yoast's code, reset OPcache. The resets were spread over a couple of hours as the run worked through the sites.

How big the table had grown by the first outage I worked out after the fact. The OOM killer report has each process's total_vm in 4 KB pages. A fresh worker that has not served a request yet matches the master, while a worker after its first request is heavier by the table plus a little heap:

# journalctl -k -S '2026-10-12 14:11:15' -U '2026-10-12 14:11:22' | grep -E 'tgid|php-fpm8.4$' | sort -k9 -n | head -9
Oct 12 14:11:21 srv kernel: [  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Oct 12 14:11:21 srv kernel: [3522917]    33 3522917   315330     3443     2552      763       128   258048      179             0 php-fpm8.4
Oct 12 14:11:21 srv kernel: [3522923]    33 3522923   315330     3474     2656      818         0   245760      243             0 php-fpm8.4
Oct 12 14:11:21 srv kernel: [3522941]    33 3522941   315330     3863     2656     1076       131   262144      243             0 php-fpm8.4
Oct 12 14:11:21 srv kernel: [3522944]    33 3522944   315330     3604     2656      948         0   245760      243             0 php-fpm8.4
Oct 12 14:11:21 srv kernel: [3532685]     0 3532685   315330     4738     2819      767      1152   290816       64             0 php-fpm8.4
Oct 12 14:11:21 srv kernel: [3522852]    33 3522852   436387   114225   112350      849      1026  1155072      244             0 php-fpm8.4
Oct 12 14:11:21 srv kernel: [3522859]    33 3522859   436387   125260   123359      875      1026  1241088      243             0 php-fpm8.4
Oct 12 14:11:21 srv kernel: [3522866]    33 3522866   436387   126088   123551      934      1603  1245184      243             0 php-fpm8.4
Enter fullscreen mode Exit fullscreen mode

The master (uid 0) and four fresh workers are at 315,330 pages, the workers after a request start at 436,387. The difference is 121,057 pages, about 473 MB: the table was nearly full size already at the first outage. Later reports give about 476 MB the next morning and about 496 MB that evening, after the mass Yoast 28.7 update across 5,478 sites. That is what smaps showed later.

The mass update on 5,478 sites added less to the table than the catch-up on 467. The sudo journal shows why: after installing, cron immediately activates the plugin.

# journalctl _COMM=sudo -S '2026-10-13 11:57:00' -U '2026-10-13 11:57:12' | grep '<site>' | grep -o 'COMMAND=.*'
COMMAND=/usr/local/bin/wp --path=/var/www/<site> core is-installed
COMMAND=/usr/local/bin/wp --path=/var/www/<site> plugin is-installed wordpress-seo
COMMAND=/usr/local/bin/wp --path=/var/www/<site> plugin get wordpress-seo --field=version
COMMAND=/usr/local/bin/wp --path=/var/www/<site> plugin is-active wordpress-seo
COMMAND=/usr/local/bin/wp --path=/var/www/<site> plugin is-active wordpress-seo --network
COMMAND=/usr/local/bin/wp --path=/var/www/<site> plugin install wordpress-seo --version=28.7 --force
COMMAND=/usr/local/bin/wp --path=/var/www/<site> plugin activate wordpress-seo
Enter fullscreen mode Exit fullscreen mode

wp plugin activate loads WordPress with the new Yoast code, so the upgrade and the reset run in the CLI. And in the CLI, OPcache is off (opcache.enable_cli defaults to 0), so such a reset never reaches FPM. It only gets into FPM if a request hits the site in the few seconds between install and activate. The manual run had no such step.

The same update of the same plugin turned out harmless or dangerous depending on which command came next.

Yoast is not the villain here. They even ship a filter to turn the reset off, and they silence the error for a disabled API with @ in advance. The problem is in PHP: a cache reset should not permanently inflate every future process.

How it played out on the server

The same day a theme update and a Polylang update went out across the sites. Those are new file versions that also need compiling, and the cache is full. More workers are needed, and each new one weighs half a gigabyte. At the moment of the first outage there were 144 workers in memory at about 500 MB of their own memory each, plus a 29 GB database, around 100 GB together. In the same report the kernel counted 99 GB of anonymous memory (active_anon plus inactive_anon), which matches.

The pool had hit 200 workers before, 39 times since the end of June, and nothing went down: two hundred normal workers fit in memory. Two hundred inflated ones are about 100 GB on a 125 GB server that also runs a database with a 32 GB buffer.

The fix

I made both config changes first and then did a single reload. restrict_api does not apply without an FPM restart, and the master reads process_control_timeout at startup. That way there is one restart, and only what breaks during it breaks.

1. Forbid OPcache resets from web requests

File /etc/php/8.4/fpm/conf.d/99-opcache-restrict-api.ini:

; forbid opcache_reset()/opcache_invalidate() from web requests.
; Every opcache_reset() under live traffic permanently inflates every new
; php-fpm worker (php-src issue #23859); updated files are still picked up
; via opcache.validate_timestamps=1 / revalidate_freq=0.
opcache.restrict_api = /nonexistent
Enter fullscreen mode Exit fullscreen mode

opcache.restrict_api allows the OPcache API functions only to scripts whose path starts with the given string. A non-existent path means "nobody". Yoast silences the error, and WordPress core checks restrict_api itself in wp_opcache_invalidate() and does nothing. OPcache still picks up updated files: with validate_timestamps=1 and revalidate_freq=0 it checks the file modification time on every access.

Two caveats. First, it also closes opcache_get_status(), opcache_get_configuration() and opcache_is_script_cached() to web scripts (the validate_api_restriction() check in zend_accelerator_module.c). An OPcache status page and my probe will not get stats anymore. If you need monitoring, set restrict_api to the prefix of a service directory, for example /usr/local/lib/opcache-status/, and keep the status script there. Second, it does not cover restarts caused by the cache running out of memory or hash overflow, and those inflate the table just the same. So a reload after mass updates is still needed.

If you don't want to close the whole API, Yoast has a targeted option, a one-line mu-plugin:

add_filter( 'Yoast\WP\SEO\should_invalidate_opcache', '__return_false' );
Enter fullscreen mode Exit fullscreen mode

I did not deploy this one on the server, I am quoting it from the filter code. Also, Yoast is not the only one that can call opcache_reset(), while the PHP-level restriction covers everyone at once.

2. So a reload doesn't cut off requests

# sed -i 's/^;process_control_timeout = 0$/process_control_timeout = 10s/' /etc/php/8.4/fpm/php-fpm.conf
# php-fpm8.4 -t
[15-Oct-2026 18:14:14] NOTICE: configuration file /etc/php/8.4/fpm/php-fpm.conf test is successful
Enter fullscreen mode Exit fullscreen mode

On reload the master sends workers SIGQUIT: finish your request and exit. After process_control_timeout it sends SIGTERM, a second later SIGKILL, and then re-executes itself (fpm_process_ctl.c). With 0, SIGTERM goes out immediately, and requests in flight at that second get cut off. With 10 seconds, everything shorter than that gets to finish. The catch: the master reads this setting at startup, so my own reload still ran with the old value and did cut off one request.

3. Reload

# systemctl reload php8.4-fpm
Enter fullscreen mode Exit fullscreen mode

In Debian's unit, reload is kill -USR2 to the master. In the FPM log it looks like this:

[15-Oct-2026 18:14:41] NOTICE: Reloading in progress ...
[15-Oct-2026 18:14:41] NOTICE: reloading: execvp("/usr/sbin/php-fpm8.4", {"/usr/sbin/php-fpm8.4", "--nodaemonize", "--fpm-config", "/etc/php/8.4/fpm/php-fpm.conf"})
[15-Oct-2026 18:14:41] NOTICE: using inherited socket fd=8, "/run/php/php8.4-fpm.sock"
[15-Oct-2026 18:14:41] NOTICE: fpm is running, pid 3532685
[15-Oct-2026 18:14:41] NOTICE: ready to handle connections
Enter fullscreen mode Exit fullscreen mode

The master replaces itself via execvp, the PID stays the same, it hands the socket to itself, and the OPcache shared memory is created fresh. The same process the next day:

# ps -o pid,etimes,cmd -p 3532685
    PID ELAPSED CMD
3532685 1756541 php-fpm: master process (/etc/php/8.4/fpm/php-fpm.conf)
Enter fullscreen mode Exit fullscreen mode

ELAPSED is twenty days, a reload does not reset it. Half a minute after the reload I looked at a new worker:

VmRSS:    171236 kB
RssAnon:       21220 kB
RssShmem:     129092 kB
Enter fullscreen mode Exit fullscreen mode

21 MB of its own memory instead of 521.

4. nginx should come back on its own

# systemctl show nginx -p Restart -p RestartUSec -p OOMPolicy -p DropInPaths -p ActiveState -p NRestarts
Restart=no
RestartUSec=100ms
NRestarts=0
OOMPolicy=stop
ActiveState=active
DropInPaths=/etc/systemd/system/nginx.service.d/limits.conf
Enter fullscreen mode Exit fullscreen mode

I added a drop-in:

# cat /etc/systemd/system/nginx.service.d/restart.conf
[Service]
Restart=on-failure
RestartSec=5s
# systemctl daemon-reload
# systemctl show nginx -p Restart -p RestartUSec
Restart=on-failure
RestartUSec=5s
Enter fullscreen mode Exit fullscreen mode

systemd treats an OOM kill as a failure, in the journal above that is Failed with result 'oom-kill'. According to the documentation, Restart= applies after such a result, and in the code of systemd 257, which is on this server, on-failure fires on any result except success (service.c). For MariaDB this path is visible in the journal:

# journalctl -u mariadb -S '2026-10-15 16:27:45' -U '2026-10-15 16:29:50' | head -6
Oct 15 16:27:48 srv systemd[1]: mariadb.service: A process of this unit has been killed by the OOM killer.
Oct 15 16:27:49 srv systemd[1]: mariadb.service: Main process exited, code=killed, status=9/KILL
Oct 15 16:27:49 srv systemd[1]: mariadb.service: Failed with result 'oom-kill'.
Oct 15 16:27:49 srv systemd[1]: mariadb.service: Consumed 4min 59.519s CPU time, 17.4G memory peak, 24K memory swap peak.
Oct 15 16:27:54 srv systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 36.
Oct 15 16:27:54 srv systemd[1]: Starting mariadb.service - MariaDB 11.8.6 database server...
Enter fullscreen mode Exit fullscreen mode

This is a safety net, not a fix.

The next day I closed the other half of the problem. With OOMPolicy=stop, killing a single worker still stops the whole service; in the nginx journal above that took 20 seconds, plus five to come back. So OOMPolicy=continue went into the same restart.conf, and PHP-FPM got its own drop-in:

# cat /etc/systemd/system/php8.4-fpm.service.d/oom-policy.conf
[Service]
OOMPolicy=continue
# systemctl daemon-reload
# systemctl show nginx php8.4-fpm -p Id -p OOMPolicy -p MainPID
MainPID=1323957
OOMPolicy=continue
Id=nginx.service

MainPID=3532685
OOMPolicy=continue
Id=php8.4-fpm.service
Enter fullscreen mode Exit fullscreen mode

Now a killed worker is only logged, the service keeps running, and its master starts a new worker. The services did not need a restart for this, the MainPIDs are the same.

The next day

# journalctl -k --since '2026-10-15 18:15' | grep -c 'Killed process'
0
# ps -o pid,etimes,rss,cmd -C php-fpm8.4 --sort=etimes | head -5
    PID ELAPSED   RSS CMD
1023872       8 82848 php-fpm: pool www
1019135      32 97032 php-fpm: pool www
1019134      32 91956 php-fpm: pool www
1019125      36 67316 php-fpm: pool www
# free -h
               total        used        free      shared  buff/cache   available
Mem:           125Gi        33Gi       1.4Gi       966Mi        92Gi        91Gi
Swap:          1.0Gi       892Mi       131Mi
Enter fullscreen mode Exit fullscreen mode

Not a single OOM since the reload. Newborn workers weigh 66-95 MB, and that includes the OPcache shared memory. This is what the largest memory chunks of a worker look like now:

# pmap -x 1019135 | sort -k3 -n | tail -4
00007f7ab9a00000    6144    6144    6144 rw---   [ anon ]
00007f7abb400000    6144    6144    6144 rw---   [ anon ]
0000556e8d400000 1048576   33196   33196 rw-s- zero (deleted)
total kB         1300164   99408   78472
Enter fullscreen mode Exit fullscreen mode

The biggest one is the 1 GB OPcache shared memory, of which this worker actually touched 32 MB. No more anonymous chunks of hundreds of megabytes.

There is a trap here that I nearly fell into myself. My monitoring script still showed about 70 GB for PHP:

2026-10-16 10:21:45 ... php=70679MB php_procs=30 db=29002MB avail=93802MB
Enter fullscreen mode Exit fullscreen mode

It takes memory from the service cgroup, and the cgroup also counts the cache of files PHP has read:

# grep -E '^(anon|file|shmem) ' /sys/fs/cgroup/system.slice/php8.4-fpm.service/memory.stat
anon 1225437184
file 59738320896
shmem 1009795072
Enter fullscreen mode Exit fullscreen mode

PHP's own memory is 1.1 GB. file is 55.6 GB of page cache that the kernel will give back on demand, and it already includes the 0.9 GB of OPcache shared memory: in cgroup v2, shmem is counted inside file, so you can't add them up. The rest of the gap to 70 GB is most likely kernel memory; I did not dig into it separately.

A month later

I went back to the server on 16 November. Since the fix: zero OOM kills in a month. PHP-FPM has been reloaded three times since then, and none of those reloads cut off a request: there is no "upstream prematurely closed" or "connection reset" from the FPM socket in the nginx error log in those minutes, so process_control_timeout now does its job. A worker that has been alive for ten minutes holds 40 MB of its own memory. The nginx restart policy is still untested in real life, simply because nothing has killed nginx since.

The issue itself is still open and still in Needs Triage, with no patch. In November one more person wrote that they hit the same problem.

Before and after

Before After
Own memory of a fresh worker (RssAnon) 521 MB 21 MB
RSS of newborn workers 581-591 MB 66-95 MB
Largest anonymous chunk 496 MB 6 MB
Fresh worker heavier than master (total_vm, VSZ) by 473 MB, at first outage by 14 MB, after 19 hours
OOM kills 82 in four days 0 in a month
opcache.restrict_api empty /nonexistent
process_control_timeout 0 10s
nginx Restart no on-failure, 5 s
OOMPolicy for nginx and PHP-FPM stop continue

What I didn't touch

One pool for 5,500 sites and a full 1 GB OPcache with a 35% hit rate is a separate job: cache size, max_accelerated_files, splitting into pools. That is not something you do at night on a falling server, and it has nothing to do with the inflation, even though it makes every request more expensive.

Transparent Huge Pages in always mode: of the 508,032 KB chunk, 505,856 KB sit in huge pages. Not related to the cause. The table is zeroed on every request, so even without THP it would be fully resident.

A memory ceiling for PHP via cgroup. With OOMPolicy=continue it can be set without fear that the first killed worker stops all of FPM, but that is the next job.

Check your own server

This applies to any server where many sites share one PHP-FPM master, and not only WordPress: any application that calls opcache_reset() under load will do.

  1. Cache restarts. From FPM itself, not from the console: a small script in the pool, like the probe above, with opcache_get_status(false). Look at manual_restarts, oom_restarts and start_time. If restrict_api is set, the script has to live under the allowed prefix.

  2. Own memory of young workers. Specifically RssAnon, not RSS: RSS also includes the OPcache shared memory.

ps -o pid,etimes,rss -C php-fpm8.4 --sort=etimes | head
grep RssAnon /proc/PID/status
Enter fullscreen mode Exit fullscreen mode

A process alive for a few seconds with hundreds of megabytes of its own memory? Keep going.

  1. What such a worker holds in memory:
pmap -x PID | sort -k3 -n | tail
Enter fullscreen mode Exit fullscreen mode

A nameless [ anon ] of hundreds of megabytes, the same in every worker and absent in the master, with a size that is a multiple of 32 KB plus 4 KB. That is the one.

  1. If the server has already gone down, you can estimate the table size after the fact from the OOM killer report: total_vm of workers that served a request minus the master's total_vm, in 4 KB pages.

  2. Who calls the reset. grep over one copy of every plugin, the themes and mu-plugins:

grep -rn opcache_reset wp-content/plugins wp-content/themes wp-content/mu-plugins
Enter fullscreen mode Exit fullscreen mode
  1. What runs after plugin updates. If the last command doesn't load WordPress (wp plugin update at the end of a script, file updates via git or rsync), the plugin's upgrade code will be run by the first visitor, which means by FPM. If install is followed by wp plugin activate or any command that loads the site, the reset stays in the CLI.

  2. What systemd will do with a killed process:

systemctl show nginx php8.4-fpm -p Restart -p OOMPolicy
Enter fullscreen mode Exit fullscreen mode

With OOMPolicy=stop, the default, killing any process of the service stops all of it: in the nginx journal above it took 20 seconds from one process being killed to the service stopping. With OOMPolicy=continue the kill is only logged, the service keeps running, and its master starts a new worker. For nginx and PHP-FPM, continue together with Restart=on-failure in case the master itself gets killed makes sense.

  1. If you found it: systemctl reload for PHP-FPM right away, then opcache.restrict_api or the plugin filter, process_control_timeout, and until the bug in PHP is fixed, a reload after every mass plugin update. That is one line at the end of the update script.

Takeaway

If a PHP process is heavy from the first second of its life, don't look for a leak in the code, look at what it gets at birth. And when a client says "DDoS", first check whether the server would have handled the same traffic a week ago.

Links

Originally published in Russian on Habr.

Top comments (1)

Collapse
 
wbcomdesigns profile image
Wbcom Designs 💡 •

Great writeup. "Blocked the scanner, sites came back, two days later everything falls over" is exactly how a wrong first diagnosis feels. The fact that a plugin update triggering cache resets could inflate every new worker is scary at 5,500 sites. Did you end up rate-limiting opcache_reset calls, or pinning the PHP build until the fix landed?