DEV Community: Mahmut Gündüzalp

PHP2Shell (CVE-2026-109618 / CVE-2026-109619): A Practical Patch, Mitigation and Incident Checklist

Mahmut Gündüzalp — Wed, 21 Oct 2026 22:04:44 +0000

On 21 October 2026 at 14:00 UTC the PHP project shipped fixes for two critical bugs in the PHP engine itself, nicknamed PHP2Shell. Both allow unauthenticated remote code execution over plain HTTP. Details leaked a day early, the release was pulled forward from 22 October, and many organisations took PHP sites offline as a precaution while packages were still being built.

This post is a working checklist: what is affected, how to update on common distributions, panels and containers, how to confirm the fix is actually running, what temporary mitigations do and do not cover, and what to look for if you could not patch in time. Everything here comes from the official advisories and vendor notes listed at the end. Where sources disagree, I say so.

What the two bugs are

PHP aborts a request when it hits memory_limit. If that abort happens at the wrong moment, internal state is left half-initialised or still points at freed memory, and the next request handled by the same worker uses it.

CVE-2026-109618 ("RCE via OOM corruption", GHSA-w8hw-w9fx-j79w). PHP leaves the request with longjmp() on memory exhaustion. If request memory is only partly initialised at that point, shutdown processes invalid state. Several multipart parsing issues (no usable content length, unlimited headers per part, nested empty-file checks) can push a request into OOM while staying under post_max_size.
CVE-2026-109619 ("RCE via stale global state", GHSA-6wmf-4rv5-p3rp). Request-scoped global state can outlive the request, and some code runs before normal request startup and reads it: a heap use-after-free. The confirmed paths are the session upload progress lifecycle (mod_php and FPM) and, on PHP 8.5, fatal_error_backtraces.

The trigger is a crafted multipart/form-data request. No login is needed, and the application code does not have to run at all. Researchers also described a memory disclosure path where reclaimed heap content comes back in the PHPSESSID response cookie, even if the app never calls session_start(). That leaked memory can hold other users' passwords, session tokens and API keys.

A note on CVE mapping: Wiz and Sansec group the upload-progress bugs under 109618, while the official PHP advisories put session upload progress under 109619. Use the GHSA advisories as the reference.

Severity: CVSS 4.0 9.2 per the PHP advisory. The 10.0 that circulated with the leak is wrong according to the PHP Foundation. NVD shows CVSS 3.1 9.8 and Red Hat rates it CVSS 3 8.1, so you will see all three numbers.

Your app framework updates do not fix this. The bug is in the interpreter. A CMS, plugin or framework update leaves you exactly as exposed as before. You have to update PHP on the server.

Patch matrix

Branch	Status
8.5.x	Fixed in 8.5.12
8.4.x	Fixed in 8.4.27
8.3.x	Fixed in 8.3.36
8.2.x	Fixed in 8.2.35
8.1 / 8.0 / 7.4	No official release. Minimal source backports were given to distributors, so the fix depends on your vendor's package
5.4 – 7.3	End of life. No fix from the PHP project. Some third-party repositories ship their own patches

The affected code path goes back to PHP 5.4.0 (2012).

Distribution packages backport. The version string often stays old. Ubuntu 24.04's fixed package is 8.3.6-0ubuntu0.24.04.13, not 8.3.36. Compare package revisions against your vendor's advisory, not upstream version numbers.

Which SAPI is exposed

SAPI	Risk
Apache mod_php	Highest. Full RCE over HTTP confirmed with default settings
PHP-FPM	High. Upload-progress use-after-free confirmed. The advisory says full OOM exploitation was not shown, while the Foundation blog says RCE via application code paths was confirmed
LiteSpeed LSAPI (lsphp), FrankenPHP	Possibly affected based on source analysis, not confirmed
php-cgi	Affected (anything that serves web requests)
CLI	No web exposure described if it does not serve HTTP

Updating, per platform

The general rule is the same everywhere: update the package, restart every PHP worker, verify over HTTP. Installing the package alone proves nothing. Control-panel PHP builds are usually separate from the OS packages, so apt or dnf will not touch them.

Debian

Debian 13 (trixie): php8.4 fixed in 8.4.27-1~deb13u1 (DSA-6574-1).
Debian 12 (bookworm): php8.2 reported fixed in 8.2.35-1~deb12u1. One source still showed bookworm as vulnerable at the time, so check the Debian security tracker before you trust it.

sudo apt-get update
sudo apt-get install --only-upgrade 'php8.2*' libapache2-mod-php8.2
dpkg -l | grep php

The Debian changelog may only say "New upstream version", so check the package version itself.

Ubuntu

22.04: 8.1.2-1ubuntu2.28 · 24.04: 8.3.6-0ubuntu0.24.04.13 · 26.04: 8.5.4-0ubuntu1.5 (USN-8948-1)
18.04 / 20.04: via Ubuntu Pro ESM (USN-8948-2)

zgrep -E 'CVE-2026-10961[89]' /usr/share/doc/php8.3-fpm/changelog.Debian.gz

Third-party Debian/Ubuntu repositories

packages.sury.org: all four supported branches were fixed on amd64 first; arm64 lagged at the last check. Updates for branches back to 5.6 were reported.
ppa:ondrej/php: all four branches fixed on amd64 and arm64 by 21:22 UTC on 21 October. Some mirrors served 8.3.35 for a while that day. If yours lags, use ppa.launchpadcontent.net directly.

RHEL, AlmaLinux, Rocky

AlmaLinux 8/9/10: all PHP streams in production repositories from 21:15 UTC on 21 October.
Red Hat: check the CVE page on access.redhat.com for your stream. I could not verify an erratum number from a primary source.

sudo dnf upgrade 'php*'
dnf updateinfo list --security | grep php
rpm -q php-common

Remi repository

Sources conflict here. The PHP announcement and some vendors said Remi would not ship on 21 October. A later status page listed fixed 8.2–8.5 builds for RHEL 8/9/10 and best-effort patches for 7.4/8.0/8.1. Run sudo dnf clean metadata, update (dnf upgrade 'php84-*' for SCL packages), and check the installed version.

cPanel (EasyApache 4)

ea-php82 to ea-php85 are fixed, with backports for ea-php74/80/81. Update via WHM > EasyApache 4 or:

dnf upgrade 'ea-php*'     # or: yum update 'ea-php*'

Then restart Apache and PHP-FPM.

Plesk

plesk-php packages are fixed on all four supported branches. EOL branches depend on builds dated 21 October or later. The panel's own PHP has a separate hotfix. Windows fixes were behind Linux.

plesk installer update

DirectAdmin (CustomBuild)

Per the vendor forum, the new build moves 8.2–8.5 to the fixed releases and adds patches for 8.1/8.0/7.4. A "failed hunk" on CentOS 7.9 with the 8.1 patch was reported.

da build update && da build php

LiteSpeed lsphp packages

Update lsphpXY from the LiteSpeed repository, then kill the PHP processes. Restarting the web server does not restart detached lsphp workers.

sudo apt update && sudo apt install --only-upgrade 'lsphp8*'   # Debian/Ubuntu
sudo dnf upgrade 'lsphp*'                                      # RHEL family
sudo pkill lsphp
apt-cache policy lsphp83    # check the candidate version first

On the day, some LiteSpeed repositories were still serving old builds for hours. Check the candidate version before you assume the update did anything. Update OPcache together with the core; a new core with an old OPcache module is an incomplete fix. If your control panel runs its own lsphp binary for the panel UI, that one needs updating too.

Docker official images

All php:8.x.y official images are fixed on amd64 and arm64. The 8.5.12RC1 and 8.4.27RC1 tags are not the fix. Pull the new tag, rebuild your images with docker build --pull, and recreate the containers. On Alpine, 3.24 has fixed 8.3/8.4/8.5 (and edge), 3.24 has no 8.2, and older Alpine releases will not get the fix.

Windows and macOS

windows.php.net lists fixed builds for all four branches; restart whatever serves PHP (IIS FastCGI, Apache, php-cgi). Homebrew has all four: brew upgrade php, then restart the services.

FrankenPHP, long-running workers, queues

FrankenPHP embeds PHP, so apt/dnf will not update it; you need a new FrankenPHP release or image. Restart queue workers and any long-running PHP server process. A "reload" command inside a framework does not reload the PHP binary.

Restart and verify

This is where most "patched" servers are still vulnerable.

# PHP-FPM
sudo systemctl restart php8.3-fpm        # RHEL: php-fpm

# Apache mod_php: full restart. A graceful reload kept the old libphp running.
sudo systemctl restart apache2           # or httpd

# Any process still mapping deleted (old) libraries?
sudo grep -lE ' /(usr|lib|opt).*\(deleted\)$' /proc/[0-9]*/maps
# Empty output = no process running the old binary.

# Debian/Ubuntu
sudo needrestart -b -r l

php -v only shows the CLI. For the web side, call a temporary <?php echo PHP_VERSION; page over HTTP for every PHP branch you run and delete it immediately, or read X-Powered-By if you have it enabled. On distro packages, the package revision is what counts.

Temporary mitigations and their limits

The PHP project is clear: updating is the only complete fix, and the 109619 advisory says there is no complete general configuration mitigation. If you cannot patch for a few hours, these reduce exposure. Set them in php.ini or the FPM pool with php_admin_value / php_admin_flag. ini_set() and .user.ini run too late. Restart workers afterwards and check the effective value with a real request.

Mitigation	What it covers	Limits
`session.upload_progress.enabled = 0` (default is On)	The upload-progress path of 109619	Does not fix the OOM bug (109618). The 8.5 backtrace path remains
`fatal_error_backtraces = 0` on 8.5	The 8.5 path of 109619	8.5 only
`memory_limit = -1`, not overridable by the app	Removes the predictable limit 109618 relies on	Partial. One request can eat all server memory. A container/cgroup limit is an availability guard, not a fix
Do not use `RLIMIT_AS` / `RLIMIT_DATA`		These trigger the vulnerable `longjmp()` path
Low `post_max_size`, front-end request size limits, limits on parts, headers, nesting, `max_input_vars`	109618 triggers	Exploits can stay inside the limits
`pm.max_requests = 1`	Breaks the producer/consumer chain on one worker	Large performance cost. Not a fix
Block `multipart/form-data` if the app does not need uploads	The known trigger path	Breaks uploads, some payment and contact forms, admin image upload. Test
Take the site offline / static page / GET and HEAD only	Safest stopgap (CISA, NSA, NCSC, BSI, CERT-EU)	An application "maintenance mode" does not help, PHP still parses the request
WAF rules	Known exploit patterns	Only covers traffic that passes through the WAF. If the origin is reachable directly, it does nothing. Attackers are changing payloads

; php.ini
session.upload_progress.enabled = 0
; PHP 8.5 only
fatal_error_backtraces = 0

; FPM pool
php_admin_flag[session.upload_progress.enabled] = off

Blocking multipart at the edge:

if ($http_content_type ~* "multipart/form-data") { return 403; }

RewriteEngine On
RewriteCond %{HTTP:Content-Type} multipart/form-data [NC]
RewriteRule ^ - [F]

Cloudflare turned on managed WAF rules for all plans, including free, from 03:02 UTC on 21 October. Paid customers should confirm the rules are in block mode. Some hosting-side rule sets only log on certain web servers instead of blocking, so read your vendor's note for your stack.

Avoid unofficial "fix" plugins and one-line scripts. They are not reviewed, and vendors advise against them.

Was I hit? A checklist

There is no official IOC list (file hashes, IP ranges) yet. These are behavioural signals. Finding nothing does not prove nothing happened, because older logs may not have recorded the relevant error levels. Copy your logs before you touch anything.

Logs

[ ] Unusual POST requests with multipart/form-data since 20 October, especially to URLs that have no upload form
[ ] "Allowed memory size exhausted" errors and bursts of HTTP 500 right after them
[ ] Unusually large or garbled PHPSESSID cookies in responses (the one documented observable sign of the memory leak)
[ ] Unexplained worker crashes (child exited on signal 11, segfaults)
[ ] WAF events matching the vendor's rule IDs

Processes (strongest signal)

[ ] sh, bash, python, perl, curl, wget, nc started by php-fpm, php-cgi, lsphp, httpd or apache2 (on Windows: cmd.exe, powershell.exe, certutil.exe)
[ ] Command lines with /dev/tcp/, mkfifo, fsockopen, or whoami, id, uname -a from the web user

Filesystem

# PHP files created or modified since the leak (adjust the web root)
find /var/www -name '*.php' -newermt '2026-10-20' -type f
# PHP files inside upload directories
find /var/www -path '*upload*' -name '*.ph*'

[ ] Short or obfuscated files with eval(, base64_decode(, gzinflate(, assert(, system(, passthru(, shell_exec(
[ ] Executables or PHP files in /tmp and /dev/shm
[ ] New cron jobs, new admin accounts (system, panel, CMS), changed authorized_keys
[ ] New .htaccess, .user.ini or auto_prepend_file entries

If anything looks wrong, assume compromise

[ ] Isolate the server, keep web and PHP logs
[ ] Rotate secrets, because of the memory leak: database passwords, API keys in .env, payment provider keys, SMTP passwords, JWT/app secrets, OAuth tokens, TLS private keys (reissue and revoke)
[ ] Invalidate all user sessions, change admin passwords, enforce 2FA
[ ] Rebuild from a clean backup if needed
[ ] Check your legal breach-notification deadlines if personal data was involved (72 hours under GDPR, and the same under Turkey's KVKK)

Still on PHP 8.1 or older?

CISA's guidance is blunt: if you cannot patch now, or you run PHP 8.1 or older, take those systems offline until they are patched or upgraded. A practical order:

Today: inventory every PHP runtime: servers, containers, staging, the control panel's own PHP, appliances with embedded PHP.
Within hours: apply a vendor backport if one exists (Ubuntu 22.04's 8.1, Ubuntu Pro ESM, cPanel ea-php74/80/81, DirectAdmin 7.4/8.0/8.1, the Sury repository, Plesk builds from 21 October). If none exists, take it offline or put it behind a VPN. At minimum: disable upload progress, block multipart, and use a WAF, knowing all three are partial.
Days: move the application to a supported branch, ideally 8.3 or 8.4. Test on staging first: removed functions, plugin compatibility, and whether any code encoder you rely on supports the new branch.
Software that cannot move: turn it into a static archive or make it reachable only from the internal network.
Long term: a supported-branch policy (each PHP branch gets two years of active support plus two years of security fixes), automatic security updates, a patch procedure that includes restarting workers, and tracking vendors who embed PHP in their products.

Two scams riding on this

Fake "$PHP2SHELL" token. Fake memecoin promotions using the PHP2Shell name are circulating on X. Pushing a fake token through compromised or impersonating accounts is a known pattern. There is no token. Do not connect a wallet.
Fake proof-of-concept repositories. GitHub and exploit aggregators have many "PHP2Shell PoC" repos. None has been confirmed as working by the PHP security team, Huntress or The Hacker News. Repos like this are a known way to deliver malware to researchers and admins. If you must look, do it in an isolated VM, never on a server.

Exploitation status (as of 22 October)

Reports disagree. Cloudflare and watchTowr reported exploitation attempts against their sensors and attackers modifying payloads to get around rules. Patchstack's own telemetry showed probing, not breaches. CISA, BSI and Huntress had not confirmed exploitation as of the evening of 21 October, and the PHP security team found no working public exploit at release time. Neither CVE was in the KEV catalogue at that point. Patch details and the diff are public, so this can change quickly. Treat unpatched servers as exposed.

Sources

PHP advisory CVE-2026-109618: https://github.com/php/php-src/security/advisories/GHSA-w8hw-w9fx-j79w
PHP advisory CVE-2026-109619: https://github.com/php/php-src/security/advisories/GHSA-6wmf-4rv5-p3rp
PHP Foundation: https://thephp.foundation/blog/2026/10/21/todays-php-security-release-and-how-were-strengthening-php-security/
php.net release notes: https://www.php.net/archive/2026.php#2026-10-21-5
CISA: https://www.cisa.gov/news-events/alerts/2026/10/21/critical-vulnerabilities-php-require-immediate-patching
UK NCSC: https://www.ncsc.gov.uk/news/critical-security-issue-affecting-php
CERT-EU 2026-018: https://cert.europa.eu/publications/security-advisories/2026-018/
BSI: https://www.bsi.bund.de/SharedDocs/Cybersicherheitswarnungen/DE/2026/2026-297981-1032.html
Red Hat CVE page: https://access.redhat.com/security/cve/cve-2026-109618
AlmaLinux: https://almalinux.org/blog/2026-10-21-php2shell-cve-2026-109618-cve-2026-109619/
Plesk: https://support.plesk.com/hc/en-us/articles/44397729587351-Vulnerability-CVE-2026-109618-and-CVE-2026-109619-Remote-code-execution-in-PHP-shipped-with-Plesk
DirectAdmin forum: https://forum.directadmin.com/threads/critical-php-updates.82726/
Cloudflare: https://blog.cloudflare.com/php-vulnerabilities-waf-protection
Wiz: https://www.wiz.io/blog/php2shell-php-rce
Sansec: https://sansec.io/research/php2shell
Huntress: https://www.huntress.com/blog/php2shell-leaked-critical-php-vulnerabilities
Patchstack: https://patchstack.com/articles/php-major-vulnerability-alert-10-2026/
Per-distro patch status: https://ma.ttias.be/php-cve-2026-109618-patch/
Tenable FAQ: https://www.tenable.com/blog/php2shell-cve-2026-109618-cve-2026-109619-frequently-asked-questions-about-critical-php-remote

A Turkish summary of this incident is on our company blog at Alesta WEB (alestaweb.com). If you spot an error or a vendor update I missed, leave a comment and I will correct the post.

My photo.jpg Was a 29-Byte HTML Page: Validating Downloaded Images Before They Reach Production

Mahmut Gündüzalp — Fri, 16 Oct 2026 07:26:24 +0000

We publish a handful of technical articles every day on our company blog, and every article gets two real photographs: a cover and one inline image. The pipeline is boring on purpose. Pick photos, download them with curl, copy them to the server, insert the article, check the page.

One morning every file in the batch downloaded "successfully". curl exited with 0 and the files had the right names and the .jpg extension. Then a quick look at the folder showed something was off.

$ ls -l foto/
-rw-r--r-- 1 user user 29  cover-1.jpg
-rw-r--r-- 1 user user 29  cover-2.jpg
...
$ file foto/*
foto/cover-1.jpg: HTML document, ASCII text
foto/cover-2.jpg: HTML document, ASCII text

Twenty-nine bytes. Not a photo, an error page with a .jpg name. This post covers why it happened and the checks we added so that a file like that can't reach a live page again.

What went wrong: an ID that looks like a file name

We take photos from Unsplash, whose license allows free commercial use. Each photo has a short public ID that appears in its page URL:

https://unsplash.com/photos/4Ya_nyKge5k

The image files themselves live on a CDN, with URLs like this:

https://images.unsplash.com/photo-1542831371-29b0f74f9713?w=1600&h=900&fit=crop

Our script built the CDN URL from the short ID:

curl -s -o cover.jpg "https://images.unsplash.com/photo-${ID}?w=1600&h=900&fit=crop&fm=jpg"

It looked right, and for a while we never noticed, because we had been copying CDN URLs by hand. The first day the script used the short IDs directly, every request returned a 404 with a tiny HTML body. The short public ID (4Ya_nyKge5k) and the storage key (1542831371-29b0f74f9713) are two different identifiers. One is for people and URLs, the other is for the file store. Nothing in the format tells you that.

The fix for the URL itself is to get the real file URL from a source that knows it. With Unsplash's official API, every photo object has a urls block (raw, full, regular...) pointing at the CDN, so you never construct it yourself. (If you use the API, its guidelines also ask you to call the photo's download_location endpoint when someone downloads it.) For one-off manual work, the photo's download link redirects to the real file, and the Location header shows the storage key:

curl -sI "https://unsplash.com/photos/${ID}/download" | grep -i '^location:'

But the URL was only half the bug. The other half was that a 404 turned into a "successful" download. That part has nothing to do with Unsplash.

Why curl said everything was fine

By default curl doesn't treat an HTTP error status as a failure. A 404 or 500 is a complete, valid HTTP response. curl saves the body and exits with 0. With -s there's no progress output either, so there's nothing on screen to look at.

There are three things worth adding to almost every download in a script.

1. Make HTTP errors fail.

curl -fsS -o cover.jpg "$URL"          # -f: exit 22 on HTTP >= 400, keep no body
curl --fail-with-body -sS -o cover.jpg "$URL"   # curl 7.76+: fail, but keep the body for debugging

-S brings back error messages that -s would hide. With set -e in the script, a 404 now stops the batch.

2. Look at what the server says it sent.

read -r code type < <(curl -sS -o cover.jpg -w '%{http_code} %{content_type}\n' "$URL")
if [ "$code" != 200 ] || [[ "$type" != image/* ]]; then
  echo "bad download: $code $type ($URL)" >&2
  exit 1
fi

-w prints transfer details after the download. Checking the content type catches the case where the status is 200 but the body is a login page, a consent page or a bot check. That last one is becoming common: a CDN or WAF answers an automated client with a "checking your browser" page and a 200 or 403.

3. Look at the bytes, not the name.

The extension is a label you chose. The first bytes of the file are what it actually is. A JPEG starts with FF D8 FF, a PNG with 89 50 4E 47, WebP with RIFF....WEBP.

file --mime-type -b cover.jpg     # image/jpeg ... or text/html

A small validator we now run on every image

The shell checks catch failed downloads. Before publishing we also want to know the image is complete and has the size the page expects. This is the Python version we run over the folder before anything is uploaded:

import sys
from pathlib import Path
from PIL import Image  # pip install pillow

EXPECTED = {"-kapak.jpg": (1600, 900), "-1.jpg": (1200, 675)}
MIN_BYTES = 10_000

def check(path: Path) -> list[str]:
    problems = []
    data = path.read_bytes()
    if len(data) < MIN_BYTES:
        problems.append(f"too small ({len(data)} bytes)")
    if not data.startswith(b"\xff\xd8\xff"):
        problems.append(f"not a JPEG (starts with {data[:12]!r})")
        return problems
    try:
        with Image.open(path) as im:
            im.verify()              # structure check, catches truncated files
        with Image.open(path) as im:
            im.load()                # full decode
            size = im.size
    except Exception as e:
        return problems + [f"cannot decode: {e}"]
    for suffix, want in EXPECTED.items():
        if path.name.endswith(suffix) and size != want:
            problems.append(f"size {size}, expected {want}")
    return problems

bad = 0
for p in sorted(Path(sys.argv[1]).glob("*.jpg")):
    issues = check(p)
    if issues:
        bad += 1
        print(f"FAIL {p.name}: {'; '.join(issues)}")
print(f"{bad} bad file(s)")
sys.exit(1 if bad else 0)

A few notes from using it:

verify() has to be followed by a fresh open(). After verify() the image object can't be used for anything else.
The size threshold is crude, but useful. A 1600x900 photo at quality 80 is usually well above 100 KB. Anything under 10 KB is almost always an error page, a placeholder, or a blank image.
Checking dimensions catches CDN parameters that were ignored. If fit=crop doesn't apply, you get the original aspect ratio, and the layout jumps.

Then check the live URL, not the local file

A correct file on disk can still be broken on the page. Since that morning we have hit two more ways for a good image to disappear after upload.

Line endings in file names. A file list produced by a Windows script and consumed by a bash loop carried a trailing \r into every file name. The files were on the server, ls showed them, and every URL returned 404, because the real name was cover.jpg\r. This finds them:

ls | grep -c $'\r'      # should print 0

The image goes through more than one URL. Our site serves images through an optimizing loader endpoint that resizes on the fly, not the raw file. The raw file URL can be 200 while the resized version fails. So the post-publish check requests both, and a headless browser loads the article and counts images that didn't decode:

// Playwright, after page.goto(...)
const broken = await page.evaluate(() =>
  [...document.querySelectorAll("main img")]
    .filter(img => img.complete && img.naturalWidth === 0).length
);

naturalWidth === 0 on a completed image means the browser got something it couldn't decode. That's exactly what a 29-byte HTML file served as image/jpeg looks like.

Takeaways

An HTTP error is not a curl error. Use -f or --fail-with-body, and -S with -s.
Check %{http_code} and %{content_type} for every download in a script. Bot-check pages often come back with a 200.
Trust magic bytes, not extensions: file --mime-type, or the first three bytes of a JPEG.
Decode the image before publishing it. Pillow's verify() plus load() catches truncated and fake files.
A public ID in a URL isn't necessarily the storage key. Take file URLs from the API response instead of building them.
After upload, test the URL the visitor actually loads, including any resizing endpoint, and count undecodable images in a real browser.

None of this is clever. It is the set of checks we wish we had from the first day. We run this pipeline for the blog at Alesta WEB (alestaweb.com). The morning it failed, nothing reached a reader, mostly because a check looked at the bytes and not the exit code.

Instant ADD COLUMN, Until a FULLTEXT Index Shows Up: A MariaDB Migration That Froze Writes

Mahmut Gündüzalp — Mon, 12 Oct 2026 20:21:16 +0000

Adding a nullable column to an InnoDB table is supposed to be a non-event. On MariaDB 10.3+ and MySQL 8.0+ it is usually an instant operation: a metadata change, no data copied, done in milliseconds no matter how big the table is.

Last week we shipped a release of a news CMS that added a few columns to its articles table. On most installations the migration finished before anyone noticed. On the larger ones, the article table (a few hundred MB) was rebuilt from scratch, the release took minutes instead of seconds, and every page view that tried to bump a view counter sat waiting until the rebuild finished.

The difference between the two groups was one line in the table definition:

FULLTEXT KEY ft_search (title, body)

This post is the short version of what happened, a reproduction you can run yourself, and the two habits that would have made it a non-event.

The setup

The schema looked roughly like this (names simplified):

CREATE TABLE articles (
  id     INT AUTO_INCREMENT PRIMARY KEY,
  title  VARCHAR(200),
  body   TEXT,
  hits   INT DEFAULT 0,
  -- ...
  FULLTEXT KEY ft_search (title, body)
) ENGINE=InnoDB;

The migration was written the way most migrations are written, one statement per change:

ALTER TABLE articles ADD COLUMN source_url VARCHAR(255) NULL;
ALTER TABLE articles ADD COLUMN is_sponsored TINYINT NOT NULL DEFAULT 0;
ALTER TABLE articles ADD COLUMN updated_by INT NULL;

Every article page runs something like UPDATE articles SET hits = hits + 1 WHERE id = ?. That detail matters later.

Reproducing it

I rebuilt the situation on a local MariaDB instance with two identical tables of 200,000 rows each (about 320 MB). The only difference: a1 has a FULLTEXT index, a2 does not.

CREATE TABLE a1 (
  id INT AUTO_INCREMENT PRIMARY KEY,
  title VARCHAR(200), body TEXT, hits INT DEFAULT 0,
  FULLTEXT KEY ft (title, body)
) ENGINE=InnoDB;

CREATE TABLE a2 (
  id INT AUTO_INCREMENT PRIMARY KEY,
  title VARCHAR(200), body TEXT, hits INT DEFAULT 0
) ENGINE=InnoDB;

First, ask the server what it is willing to do instead of letting it choose:

ALTER TABLE a2 ADD COLUMN c1 INT NULL, ALGORITHM=INSTANT;
-- Query OK

ALTER TABLE a1 ADD COLUMN c1 INT NULL, ALGORITHM=INSTANT;
-- ERROR 1845 (0A000): ALGORITHM=INSTANT is not supported for this operation.
-- Try ALGORITHM=INPLACE

ALTER TABLE a1 ADD COLUMN c1 INT NULL, ALGORITHM=INPLACE, LOCK=NONE;
-- ERROR 1846 (0A000): LOCK=NONE is not supported.
-- Reason: Fulltext index creation requires a lock. Try LOCK=SHARED

That second error is the whole story. With a FULLTEXT index on the table, adding a column is not instant. The table gets rebuilt, and because the full-text index has to be rebuilt along with it, the server needs at least a shared lock: reads keep working, writes wait.

Then I timed it, with a second connection firing a single counter update one second into each ALTER:

Operation	Time	Concurrent `UPDATE ... hits = hits + 1` waited
`a1` (FULLTEXT), two columns in one `ALTER`	34.3 s	33.3 s
`a1` (FULLTEXT), two columns in two `ALTER`s	177.1 s total	44.3 s (during the first one)
`a2` (no FULLTEXT), two columns in one `ALTER`	0.06 s	–

Same table size, same columns. Without the FULLTEXT index the change took 60 milliseconds. With it, a single counter update was blocked for more than half a minute, on a laptop with nothing else running.

On a live site that is not one blocked update. It is every page view during that window, each holding a connection while it waits. Depending on your connection limits and timeouts, that is the difference between "the release was a bit slow" and "the site returned errors for a minute".

Why FULLTEXT changes the rules

InnoDB implements full-text search with a set of auxiliary tables and a hidden FTS_DOC_ID column that ties each row to its entries in the full-text index. The instant ADD COLUMN path works by changing only metadata and leaving existing rows physically untouched. Tables with a FULLTEXT index are excluded from that path, so the server falls back to rebuilding the table. During the rebuild it also rebuilds the full-text index, and that step can't run with concurrent writes, hence LOCK=SHARED.

None of this is a bug. It is documented behaviour in both MariaDB and MySQL. The problem is that it is easy to never hit it in development, where the table has a few hundred rows and every ALTER finishes instantly regardless of the algorithm.

Mistake 1: one statement per column

Our migration had three ALTER TABLE statements, which meant three full rebuilds, three times the write freeze. The fix is boring:

ALTER TABLE articles
  ADD COLUMN source_url   VARCHAR(255) NULL,
  ADD COLUMN is_sponsored TINYINT NOT NULL DEFAULT 0,
  ADD COLUMN updated_by   INT NULL;

One statement, one rebuild. In the reproduction, two separate statements took 177 seconds total versus 34 for the combined one. You will see different numbers on your hardware, but the shape holds: rebuild cost scales with the number of statements, not the number of columns.

This is worth making a rule even for tables without FULLTEXT today. Indexes get added later, by someone else, for a search feature, and your migration habits don't get re-reviewed when that happens.

Mistake 2: letting the server pick the algorithm

When you write a bare ALTER TABLE ... ADD COLUMN, the server silently picks the cheapest algorithm it can use. If instant isn't available, it quietly does a rebuild. Nothing in the output tells you which one happened, except the time it took.

Make the expectation explicit:

ALTER TABLE articles
  ADD COLUMN source_url VARCHAR(255) NULL,
  ADD COLUMN updated_by INT NULL,
  ALGORITHM=INSTANT;

If instant is possible, nothing changes. If it isn't, the statement fails immediately with error 1845 instead of starting a multi-minute rebuild. A failed migration in a release script is annoying; a silent table rebuild on a busy table is an incident. You want the annoying one.

For a migration that is meant to be online but can't be instant, ask for the lock level you can actually tolerate:

ALTER TABLE articles ADD COLUMN x INT NULL, ALGORITHM=INPLACE, LOCK=NONE;

Same idea: fail fast if the server can't keep writes flowing, then decide consciously.

What to do when the rebuild is unavoidable

Once ALGORITHM=INSTANT has told you "no", you have a few options. None of them is free.

1. Schedule it. Run the single combined ALTER at the quietest hour. For many sites, a 30-second write pause at 4 a.m. is fine, as long as it is expected and short. Put a short lock_wait_timeout on the session so the ALTER gives up instead of queueing behind a long-running transaction:

SET SESSION lock_wait_timeout = 10;
ALTER TABLE articles ADD COLUMN ..., ADD COLUMN ...;

This protects against the other classic: the ALTER waits for a metadata lock held by some long query, and every new query on the table piles up behind the waiting ALTER.

2. Make the hot writes tolerate it. In our case the blocked writes were view counters. They don't need to hit the main table synchronously. Buffering them (a small counter table, a cache, or a queue that is flushed periodically) removes the most frequent write from the critical table. That helps far beyond migrations.

3. Drop and re-create the FULLTEXT index around the change. Drop the index, add the columns (now instant), add the index back. The column change becomes cheap, but re-creating the full-text index is itself a long operation, and search returns nothing until it completes. This only makes sense if search being unavailable for a while is acceptable and writes are not.

4. Use an online schema change tool. pt-online-schema-change and gh-ost copy the table in the background and swap it in at the end. They are built for exactly this, but they add operational complexity, need care with triggers and foreign keys, and are overkill for a 300 MB table.

A pre-flight check you can add to your release script

Before running migrations against a large installation, list the tables that can't get instant column changes:

SELECT DISTINCT s.table_name,
       ROUND((t.data_length + t.index_length) / 1048576) AS size_mb
FROM information_schema.statistics s
JOIN information_schema.tables t
  ON t.table_schema = s.table_schema AND t.table_name = s.table_name
WHERE s.table_schema = DATABASE()
  AND s.index_type = 'FULLTEXT'
ORDER BY size_mb DESC;

If a migration touches one of those tables and the table is larger than a few tens of MB, the migration needs a plan, not just a statement.

And the cheapest test of all: run the migration against a copy of your largest real dataset, not the development seed. Our development database had a few hundred articles. The rebuild there took less time than the HTTP request that triggered it.

Takeaways

Instant ADD COLUMN is the default you get used to, not a guarantee. A FULLTEXT index on the table switches it off.
Without instant, the table is rebuilt and, because of the full-text index, writes are blocked for the duration (LOCK=SHARED).
Put all column changes for a table into one ALTER TABLE.
Write ALGORITHM=INSTANT (or LOCK=NONE) explicitly so impossible expectations fail fast instead of turning into a silent rebuild.
Test migrations against production-sized data and keep frequent, low-value writes like counters off your largest tables.

We hit this while rolling out a release of the news CMS we build at Alesta WEB (alestaweb.com). The fix in the end was two lines in the migration. Finding out why the release had stalled on the big sites took considerably longer.

Green Build, Broken Site: Three Next.js 16 Failures You Only See in Production

Mahmut Gündüzalp — Thu, 01 Oct 2026 12:20:21 +0000

One Next.js 16 site, one working day, three failures. None of them showed up on the development machine. All three passed the build. Each one broke a single feature quietly while the rest of the site looked fine.

Image uploads in the admin panel returned 422 for every file.
Files uploaded after the server started returned 404 until the next restart.
Scheduled posts went live in the database but never appeared on the site.

They share one root cause: the environment you test in is not the one you ship. Below is each failure, how it showed up, the fix, and the check that would have caught it before a user did.

1. The image library that compiled to nothing

Symptom

The admin upload endpoint re-encodes every image with sharp (strip metadata, resize, convert to WebP). In production it answered every request with:

{ "hata": "Görsel işlenemedi" }

That's "image could not be processed", the message from our own catch block. The interesting part was what the catch block was hiding.

Why it only happened in production

Locally we built with next build --webpack, because a font download failed under Turbopack on that Windows machine. The server ran a plain next build, which in Next.js 16 means Turbopack.

In the Turbopack output, the static import

import sharp from "sharp";

had been replaced with (void 0). The handler compiled, the route existed, and then the first call failed at runtime.

The usual advice is to list the package in serverExternalPackages. sharp is already on Next's built-in list, and adding it explicitly made no difference in that build.

Fix

Resolve the module at runtime, from the app's own node_modules, so that the bundler has nothing to analyse statically:

import { createRequire } from "node:module";
import path from "node:path";

type Sharp = typeof import("sharp");
const MODULE_NAME = "sharp"; // a variable, not a literal: nothing to resolve at build time
let cached: Sharp | null = null;

export function loadSharp(): Sharp {
  cached ??= createRequire(path.join(process.cwd(), "package.json"))(MODULE_NAME) as Sharp;
  return cached;
}

The type import keeps full typings, and loadSharp()(buffer).webp().toBuffer() behaves exactly as before.

The check that would have caught it

Build in CI with the same bundler you deploy with. If your local build needs --webpack to work around something, that is a second build pipeline, and only one of the two runs in production. The upload was tested, but against the webpack build.

2. Uploads that 404 until the next restart

Symptom

Once uploads worked, a new one appeared: the API returned a URL like /uploads/logos/abc123.webp, the file was on disk with correct permissions, and the URL returned 404. After a process restart the same URL worked.

Why

next start serves the public/ files it found when the process started. Anything written there afterwards is invisible to the static file handler: the same request goes from 404 to 200 after nothing but a restart. The dev server is more forgiving, which is why nobody noticed.

public/ is meant for assets that ship with the build, not for user content. Most guides tell you to move uploads to object storage, and that is the right long-term answer. But if you self-host on a single server and want to keep files on local disk, there is a small fix.

Fix

Keep writing to public/uploads/, and add a catch-all route handler that serves from that directory only when the static handler misses:

// app/uploads/[...path]/route.ts
import path from "node:path";
import { promises as fs } from "node:fs";

const ROOT = path.join(process.cwd(), "public", "uploads");
const TYPES: Record<string, string> = {
  ".webp": "image/webp", ".png": "image/png", ".jpg": "image/jpeg",
  ".jpeg": "image/jpeg", ".gif": "image/gif", ".avif": "image/avif",
};

export async function GET(_req: Request, { params }: { params: Promise<{ path: string[] }> }) {
  const { path: parts } = await params;
  if (!parts.length || parts.some((p) => !p || p === "." || p === ".." || p.includes("\\") || p.includes("\0"))) {
    return new Response("Not found", { status: 404 });
  }
  const file = path.resolve(ROOT, ...parts);
  const type = TYPES[path.extname(file).toLowerCase()];
  if (!type || !file.startsWith(ROOT + path.sep)) return new Response("Not found", { status: 404 });

  try {
    const st = await fs.stat(file);
    if (!st.isFile()) return new Response("Not found", { status: 404 });
    return new Response(new Uint8Array(await fs.readFile(file)), {
      headers: {
        "Content-Type": type,
        "Content-Length": String(st.size),
        "Cache-Control": "public, max-age=31536000, immutable", // names are random and never reused
        "X-Content-Type-Options": "nosniff",
      },
    });
  } catch {
    return new Response("Not found", { status: 404 });
  }
}

Three details matter more than the rest:

Path traversal. Reject .. segments and confirm that the resolved path is still under the root. Either check alone has gaps.
No SVG. An SVG can carry script. If you need SVG uploads, serve them as Content-Disposition: attachment or from another origin.
Immutable caching is only safe because file names are random. If your uploads keep their original names and can be overwritten, drop immutable.

Files present at startup are still served by the static handler. The route only handles the rest.

The check that would have caught it

Test upload-then-fetch against a production build, without restarting in between. "Upload returns 200" is not the test. "The returned URL serves the bytes" is.

3. Scheduled posts that published but never appeared

Symptom

A cron job calls an endpoint every ten minutes. The endpoint flips scheduled posts whose time has come to published, then invalidates the posts cache tag. The database showed the posts as published. The home page, the sidebar and the "latest posts" blocks still showed the old list, indefinitely.

The cron used curl -s ... > /dev/null, so its failures went nowhere. The process log had the answer:

Error: updateTag can only be called from within a Server Action. To invalidate
cache tags in Route Handlers or other contexts, use revalidateTag instead.

Why

Next.js 16 split tag invalidation in two:

updateTag(tag): Server Actions only. Read-your-own-writes semantics, so the user who submitted the form sees the change immediately.
revalidateTag(tag, profile): works anywhere on the server, including Route Handlers, and now takes a second argument.

The endpoint called updateTag. The database write happened first, then the call threw, and the cache was never touched. Because the post was already published, the next cron run found nothing to do, so it never retried. And because data cached with unstable_cache and no revalidate time stays until its tag is invalidated, "eventually" never came.

Fix

import { revalidateTag } from "next/cache";

// updateTag only works inside Server Actions; route handlers use revalidateTag.
if (result.count > 0) revalidateTag("posts", { expire: 0 });

{ expire: 0 } expires the entries immediately. The "max" profile would serve stale data once and refresh in the background. For "the post should be visible now", expire immediately.

We verified it end to end: inserted a scheduled test post, called the endpoint (a JSON body reporting one published post, instead of a 500), requested a page and saw the post in the sidebar, then deleted the test row.

One more trap: deleting a row with raw SQL invalidates nothing. After removing the test post, it stayed in the cached sidebar until the tag was invalidated again. Any out-of-band write (a migration, a manual fix, an import script) needs a tag invalidation afterwards, or a way to trigger one.

The check that would have caught it

Don't send cron output to /dev/null. Log the response body, or at least fail loudly on a non-2xx status (curl -fsS). The endpoint had been returning 500 for every run that had work to do.

What the three have in common

Failure	Dev	Prod	What actually differed
`sharp` was `undefined`	worked	422	webpack locally, Turbopack on the server
new uploads 404	worked	404 until restart	dev server re-scans `public/`, prod doesn't
cache never refreshed	not exercised	stale forever	the code path only ran from cron

None of these is a Next.js bug in the sense of "the framework is broken". Two are documented behaviours and one is a bundler edge case with a clean workaround. All three survived because the only environment that ran the code path was production, and production was set up to stay quiet about failures.

The checklist we now run before calling a deploy done:

Build with the production bundler, no flags that the server doesn't use.
Upload a file, then fetch the returned URL without restarting.
Run every cron endpoint once by hand and read the response body.
After any manual database change, invalidate the affected cache tags.

We build news, e-commerce and restaurant POS software at Alesta WEB. These notes come from moving our own site to a new stack this week.

Two Proxies, One Comma, Every Page a 500

Mahmut Gündüzalp — Mon, 28 Sep 2026 06:29:57 +0000

A Next.js site went live behind a CDN and every single page returned 500. Not the admin area, not the login route — the home page, the about page, static-looking marketing pages. The build was green. The same build ran without a single error on the development machine.

The cause was one header with one extra value in it:

x-forwarded-proto: https, https

This post walks through why that header looks the way it does, exactly where it breaks, why the obvious fix causes a second, different failure, and the two neighbouring headers that go wrong for the same reason.

All behaviour below was re-measured afterwards in isolation, against the same library build that was deployed. Nothing is from memory.

The shape of the deployment

browser ──https──▶ CDN ──https──▶ origin web server ──http──▶ Next.js (node)

Two reverse proxies sit in front of the application: the CDN at the edge, and the web server on the origin machine that terminates TLS and forwards to the Node process on a local port.

Both of them are doing their job correctly. Each one records what it saw on the way in. The CDN saw HTTPS, so it sets X-Forwarded-Proto: https. The origin server also received HTTPS — from the CDN — so it adds its own https.

Depending on the proxy, "adds" means either a second header line or a comma-appended value. From the application's point of view it makes no difference, and that is the first thing worth measuring.

Two header lines are one string

The Fetch Headers object, which is what a Next.js middleware receives, combines repeated header lines into a single comma-separated value:

const h = new Headers();
h.append("x-forwarded-proto", "https");
h.append("x-forwarded-proto", "https");

h.get("x-forwarded-proto");
// "https, https"

So there is no way to "just read the first one" through .get(). Whatever sits between you and the client, you get the list.

Where it breaks

The site used Auth.js v5 (next-auth beta) with its auth() wrapper around the middleware. The wrapper is convenient: it resolves the session for every request and hands it to your middleware as req.auth.

To resolve the session, it builds an internal URL for its own session endpoint. When no AUTH_URL environment variable is set, that URL is assembled from request headers. The relevant part of createActionURL in @auth/core reads:

const detectedHost = headers.get("x-forwarded-host") ?? headers.get("host");
const detectedProtocol = headers.get("x-forwarded-proto") ?? protocol ?? "https";
const _protocol = detectedProtocol.endsWith(":")
  ? detectedProtocol
  : detectedProtocol + ":";
url = new URL(`${_protocol}//${detectedHost}`);

With one proxy, that produces https://example.org. With two, it produces this string:

https, https://example.org

which is not a URL. Calling it directly, against the same library build:

Input	Result
single `https`	`https://example.org/api/auth/session`
`https, https` (two header lines)	`TypeError: Invalid URL`
`https, https` (one comma list)	`TypeError: Invalid URL`
`http, https`	`TypeError: Invalid URL`

The error is thrown inside the middleware, and the middleware runs on every matched path. So it is not "auth is broken" — it is the site that is broken. Every page, including ones that never look at the session, returns 500.

The development machine never showed it because it had no proxy in front at all, let alone two.

The obvious fix, and the second failure

Reading the code above, the escape hatch is clear: if AUTH_URL is set, the header branch is never taken. Measured:

Input	`AUTH_URL`	Result
`https, https`	unset	`TypeError: Invalid URL`
`https, https`	`https://example.org`	`https://example.org/api/auth/session`

So setting AUTH_URL removes the crash. But in the wrapper, AUTH_URL does a second thing. Before your middleware runs, the request itself is rebuilt with the configured origin:

export function reqWithEnvURL(req) {
  const url = process.env.AUTH_URL ?? process.env.NEXTAUTH_URL;
  if (!url) return req;
  const { origin: envOrigin } = new URL(url);
  const { href, origin } = req.nextUrl;
  return new NextRequest(href.replace(origin, envOrigin), req);
}

Inside the application, the request used to say http://127.0.0.1:3000/.... Now it says https://example.org/.... Any later middleware that builds a rewrite from req.url — the i18n locale rewrite, in this case — now rewrites to the public address instead of the local one. A rewrite to an external origin is a proxy hop back out through the CDN, into the same middleware, which rewrites again.

The observed result on the live site was a redirect loop on /. The crash was gone; the site was still down.

What actually fixed it

The middleware never needed the full session. It needed one question answered — is there a valid admin token on this request? — to guard a handful of routes.

getToken from next-auth/jwt answers exactly that, and its implementation touches only two headers: cookie and authorization. It never parses the protocol, never parses the host, never builds a URL:

import { getToken } from "next-auth/jwt";

const token = await getToken({
  req,
  secret: process.env.AUTH_SECRET,
  secureCookie: true,
});

Replacing the auth() wrapper with getToken in the middleware fixed both failures at once. AUTH_URL stays in the environment, because the route handlers under /api/auth/* still use it — it just no longer rewrites every request passing through middleware.

The `secureCookie` line is not optional

getToken has to know the cookie's name, and the name depends on whether the session was issued over HTTPS. Measured from the library's cookie defaults:

`secureCookie`	Cookie name it looks for
`false` (default)	`authjs.session-token`
`true`	`__Secure-authjs.session-token`

Behind TLS, the login flow issues the __Secure- cookie. Without secureCookie: true, getToken looks for the other name, finds nothing, and returns null — no error, just "not logged in", forever. The flip side: with it hard-coded to true, admin login on a plain-HTTP local dev server will not work. That is the right trade-off for production, but it needs to be written down somewhere a teammate will find it.

The neighbouring headers have the same problem

Once one forwarded header is known to carry a list, the others deserve the same suspicion.

`X-Forwarded-For`: the last value is the CDN

Common advice for reading the client IP is "take the last value of X-Forwarded-For, because the client can forge the earlier ones." That advice assumes exactly one trusted proxy. With a CDN in front, the chain looks like this:

x-forwarded-for: 203.0.113.7, 198.51.100.20
                 └ client     └ CDN edge node

The last value is the CDN's edge node. Every visitor routed through that node shares it. A per-IP rate limiter or login lockout keyed on it doesn't limit one abuser — it limits everyone at once, and one failed-password burst locks the whole region out of the login form.

This was caught in review before it shipped, but it is the same class of bug. The fix is to prefer the CDN's own client-IP header — Cloudflare sends CF-Connecting-IP, other CDNs have their own equivalent — and to fall back to X-Forwarded-For only when that header is absent. Only trust the CDN header if the origin actually refuses traffic that did not come through the CDN; otherwise anyone can set it.

Canonical redirects: read the first value, or the CDN's own signal

The same middleware issued an HTTP → HTTPS canonical redirect. It now reads the CDN's visitor-scheme header where available, and otherwise takes the first element of X-Forwarded-Proto — the hop nearest the browser — after splitting on the comma:

const proto = (req.headers.get("x-forwarded-proto") ?? "")
  .split(",")[0]
  .trim();

Reading the whole string and comparing it to "https" would have been false for every request, and redirected every request to itself.

Checklist for "works on staging, 500 behind the CDN"

Count your proxies. A setup with one proxy (or none) and production with two are different environments, even with identical code.
Dump the forwarded headers from production once — x-forwarded-proto, x-forwarded-host, x-forwarded-for — and look for commas.
Grep dependencies for code that turns those headers into a URL. Any new URL(\${proto}://${host}) is a crash waiting for a second proxy.
Keep middleware small. If it only needs "is there a token", read the token; don't resolve a whole session on every request.
Know what your auth environment variables do besides the thing you set them for. AUTH_URL fixed one failure and caused another.
Never key rate limits on the last X-Forwarded-For value behind a CDN.
Test through the full chain before cutover, not against the origin port directly.

The general shape

Nothing here is a bug in the usual sense. The CDN appended correctly. The origin server appended correctly. The Fetch API combined repeated headers exactly as specified. The auth library built a URL from the headers it was given. Each piece is correct on its own; the failure is in the assumption, three layers deep, that a header which is allowed to be a list will only ever hold one item.

Headers with Forwarded in the name describe a chain. Code that reads them as a single value works until the day the chain gets one link longer — and that day is usually the day you put the site behind a CDN, in production, in front of everyone.

Written at Alesta WEB, from a production cutover where every page returned 500 until one comma was accounted for.

Six Rows Deleted. Three Decrements Applied. No Error.

Mahmut Gündüzalp — Wed, 16 Sep 2026 06:52:54 +0000

A cached counter on a listing page was wrong. Not wrong in an interesting way — just quietly, slightly too high. The kind of number nobody audits because nobody imagines it can drift.

It had been drifting for months. Across thirteen installations of the same application, the reconciliation query eventually found roughly 83,000 rows where the stored counter disagreed with the actual count. No error log entry. No failed query. No exception. Every write had returned success, every day, for months.

The cause is a single line of SQL that reads correctly, executes correctly, reports success correctly, and does not do what it says.

The query

The cleanup job removes flagged replies and decrements the per-topic counter. One statement for the delete, one for the counter:

UPDATE topics t
  JOIN replies r ON r.topic_id = t.id
  SET t.reply_count = t.reply_count - 1
WHERE r.spam = 1;

DELETE r FROM replies r WHERE r.spam = 1;

Read it out loud and it sounds right. For every spam reply, subtract one from its topic's counter. Then delete the replies.

It is not right. The UPDATE subtracts one. Total. Per topic. No matter how many replies matched.

The measurement

Three topics, nine replies, six of them flagged:

CREATE TABLE topics (
  id INT PRIMARY KEY,
  reply_count INT NOT NULL DEFAULT 0
) ENGINE=InnoDB;

CREATE TABLE replies (
  id INT PRIMARY KEY AUTO_INCREMENT,
  topic_id INT NOT NULL,
  spam TINYINT NOT NULL DEFAULT 0,
  KEY (topic_id)
) ENGINE=InnoDB;

INSERT INTO topics VALUES (1,5),(2,3),(3,1);
INSERT INTO replies (topic_id,spam) VALUES
 (1,1),(1,1),(1,1),(1,0),(1,0),
 (2,1),(2,1),(2,0),
 (3,1);

Topic 1 has three flagged replies, topic 2 has two, topic 3 has one. Counters start at 5, 3, 1. After removing the flagged replies they should read 2, 1, 0.

Run the UPDATE:

id  reply_count
1   4
2   2
3   0

Topic 1 lost one instead of three. Topic 2 lost one instead of two. Topic 3 is correct — and that is the whole problem, because topic 3 is what a hand test looks like.

Identical numbers on two different MariaDB major versions. This is not a version quirk and not a bug. It is the documented behavior of a multi-table UPDATE: each target row is visited at most once per statement, regardless of how many joined rows satisfy the condition. The engine is not iterating the join result and applying your expression once per row. It is iterating the target table and asking "does any joined row match?"

The answer is yes or no. Not three.

Why nobody catches it in testing

This is the part worth internalizing, because the trap is not the semantics — it's the shape of the evidence.

A constant assignment behaves perfectly. Change the SET from an expression to a literal:

UPDATE topics t
  JOIN replies r ON r.topic_id = t.id
  SET t.reply_count = 0
WHERE r.spam = 1;

id  reply_count
1   0
2   0
3   0

Flawless. Every topic with at least one flagged reply gets zeroed. SET status = 'archived', SET flagged = 1, SET reviewed_at = NOW() — every one of these works exactly as written, because "visit the row once" and "set it to a constant" produce the same result whether you visit once or ten times.

Multi-table UPDATE ... JOIN is a pattern developers use dozens of times and it is correct in almost every one of them. It becomes wrong the moment the right-hand side of the assignment references the column being assigned. Nothing in the syntax marks that boundary.

The client reports identical success. I ran the broken query and the correct query through the same connection and read back the driver's counters:

wrong query -> affected_rows = 3 | Rows matched: 3  Changed: 3  Warnings: 0
right query -> affected_rows = 3 | Rows matched: 3  Changed: 3  Warnings: 0

Three topics matched, three topics changed, zero warnings. Both times. The broken statement and the correct statement are indistinguishable from the application's point of view. If your job logs "updated 3 topics", that log line is true and useless.

Now put the delete next to it:

DELETE -> affected_rows = 6

Six rows deleted, three decrements applied. That discrepancy is the only signal the database ever gives you, and it appears nowhere unless you deliberately compare the two numbers. Nobody compares them, because in a healthy job they are supposed to differ — you delete N replies and touch M topics, and M is naturally smaller. The number that is wrong looks exactly like the number that is right.

The drift compounds silently. Run the cleanup three times against a topic with three flagged replies:

run 1 -> reply_count = 4
run 2 -> reply_count = 3
run 3 -> reply_count = 2

One per run, forever, in the wrong direction from the truth. A nightly job doesn't produce one visible error; it produces a slow leak. By the time somebody notices the number looks odd, there is no incident to correlate it with — no deploy, no outage, no spike. Just a figure that has been wrong for a while.

The fixes that don't work

The instinct once you understand the cause is to make the UPDATE iterate the join instead of the target. You can't. I tried the three obvious routes:

JOIN + GROUP BY            REJECTED (syntax error)
UPDATE DISTINCT            REJECTED (syntax error)
JOIN + ORDER BY + LIMIT    ACCEPTED

The first two are refused outright. The third is accepted on MariaDB and is worth a warning: it parses, it runs, it returns success, and it does not change the "once per target row" rule at all. You get a statement that looks like it addresses the problem and doesn't. (MySQL rejects ORDER BY/LIMIT on a multi-table UPDATE — so this particular false trail is engine-dependent, which is worse than if it were universal.)

There is no flag. The aggregation has to happen before the UPDATE sees the rows.

The fix

Aggregate in a derived table, join against that, and subtract the count:

UPDATE topics t
  JOIN ( SELECT topic_id, COUNT(*) AS n
           FROM replies
          WHERE spam = 1
          GROUP BY topic_id ) g
    ON g.topic_id = t.id
  SET t.reply_count = t.reply_count - g.n;

id  reply_count
1   2
2   1
3   0

Now the target table is still visited once per row — that never changed — but the value it needs is already a single number. The join produces exactly one row per topic by construction.

If the counter is derivable at all, the sturdier option is to stop doing arithmetic on it and recompute it outright:

UPDATE topics t
  SET t.reply_count = ( SELECT COUNT(*)
                          FROM replies r
                         WHERE r.topic_id = t.id
                           AND r.spam = 0 );

id  reply_count
1   2
2   1
3   0

Same result, and with a property the incremental version can never have: it is idempotent. Run it twice, run it a hundred times, the answer is the answer. An incremental counter is a running total of every code path that ever touched it, including the ones that were wrong and the ones that crashed halfway. An absolute recount has no memory of any of that. It costs more, and on a table where the counter is read far more often than it is written, that cost buys you a number that cannot drift.

Use the derived-table form when a full recount is genuinely too expensive. Use the recount when it isn't. Do not use the plain - 1.

Finding the damage you already have

If this pattern is anywhere in your codebase, the counters are already wrong and have been for a while. The reconciliation query is cheap and worth running before you fix anything, so you know the size of the problem:

SELECT t.id,
       t.reply_count             AS stored,
       COUNT(r.id)               AS actual,
       t.reply_count-COUNT(r.id) AS drift
FROM topics t
LEFT JOIN replies r ON r.topic_id = t.id
GROUP BY t.id, t.reply_count
HAVING t.reply_count-COUNT(r.id) <> 0
ORDER BY ABS(t.reply_count-COUNT(r.id)) DESC;

id  stored  actual  drift
1   4       2       2
2   2       1       1

One detail that will bite you when you write this from memory: the expression in HAVING has to be repeated in full. Referencing the drift alias fails on MariaDB with Reference 'drift' not supported (reference to group function). Same for the ORDER BY. It is a thirty-second detour that reads like a typo when you hit it at the end of a long day.

LEFT JOIN matters too. An inner join silently drops every topic whose replies were all deleted — and those are exactly the rows most likely to be wrong.

The general shape

The defect here is not SQL trivia. It is a statement whose plain-English reading and whose actual semantics diverge, where every observable signal — return code, affected row count, warning count, error log — agrees with the reading and not with the semantics.

You cannot test your way out of that with assertions on success. The only test that catches it asserts on the value: delete three replies from one topic, then check the counter went down by three. That test takes four lines. It is the difference between finding this in an afternoon and finding it in 83,000 rows.

If a counter in your schema has no test that pins its arithmetic to a real count, assume it is wrong and go measure it. It is a five-minute query and the answer is rarely zero.

Written at Alesta WEB, from a real cleanup job that had been quietly subtracting one for months.

The Socket Was Open. The Site Was Dead for Fourteen Hours.

Mahmut Gündüzalp — Mon, 07 Sep 2026 06:59:17 +0000

A site had been down since 19:54 the previous evening. Nobody reported it. The monitoring did not fire. When I finally looked, the cause was one process that had outlived its parent and was holding a socket nobody could use.

Every individual check passed. DNS resolved. Port 443 accepted connections. The TLS handshake completed against a valid certificate with two months left on it. The web server was running and serving other sites on the same machine. And every request to that one site returned 503.

The failure lived in the gap between "the process exists" and "the process works," which is a gap most health checks do not cover.

What the checks reported

I started from the outside, because the outside is cheap to measure:

Layer	Check	Result
DNS	resolves to the expected address	fine
TCP	ports 80 and 443 accept	fine
TLS	handshake, chain, expiry	fine, 68 days left
HTTP (port 80)	`GET /`	302 to the HTTPS URL
HTTPS	`GET /`	first a 20-second hang, then 503
Web server	running, other sites on the box served fine	fine

Two of those rows are the reason nobody noticed.

The port-80 row returns 302 because the HTTP-to-HTTPS redirect is a server-level rewrite. It never reaches PHP. A monitor watching http:// sees a chain that starts with a perfectly healthy 302, and if it treats a redirect as up, it reports up. Forever. The application behind it can be completely dead.

The TLS row is worse, because it looks like the most meaningful check in the list. A successful handshake proves the listener is alive and the certificate is valid. It proves nothing whatsoever about whether anything can generate a response body.

The first HTTPS request I made hung for twenty seconds and timed out. The second returned 503 in 143 milliseconds. That inconsistency is itself a clue: the server was trying to hand the request to a backend, waiting, giving up, and eventually not bothering to wait at all.

The log line that mattered

The site's own error log was useless. Its last entry was seventeen days old, because the log level was WARN and this condition never produced a warning at the virtual-host level.

The server-wide log had it, at INFO:

[INFO] [...#site] Connection idle time too long: 301 while in state: 6 ...
[INFO] [...#site] HttpExtConnector state: 0, request body sent: 0,
                  response body size: -2, response body sent: 0, attempts: 0.
[INFO] [...#site] External processor is not available.
[NOTICE] [...#site] oops! 503 Service Unavailable

External processor is not available is the whole diagnosis in five words. The web server had a request, knew which backend should handle it, and could not get a usable connection to that backend. In LSAPI terms the external processor is the PHP process pool. In PHP-FPM terms it is the pool behind the socket you point fastcgi_pass at. Same shape, same failure mode.

Note attempts: 0 and response body size: -2. It did not fail partway through a response. It never got one started.

PPID 1

The pool for this site is configured for ten children and ten concurrent connections. I expected either zero processes — pool dead, should have respawned — or ten busy ones.

There was exactly one:

  PID   PPID STAT     ELAPSED     TIME %CPU   RSS CMD
793380      1 SN      13:52:58 00:07:31  0.9  7036 lsphp

Read that line carefully, because it contains the entire bug.

PPID 1. The parent is gone. This process was adopted by init. A pool child whose master has died is an orphan, and an orphan is supervised by nothing. Nothing will restart it, nothing will kill it, nothing will notice it.

ELAPSED 13:52:58 against TIME 00:07:31. Fourteen hours of wall clock, seven and a half minutes of CPU. It was not spinning. No load-average alert was ever going to catch this.

STAT SN — interruptible sleep, low priority. The kernel confirms it:

$ grep State /proc/793380/status
State:  S (sleeping)

$ cat /proc/793380/wchan
hrtimer_nanosleep

hrtimer_nanosleep means the process is inside a timed sleep. Some PHP path called sleep() and the value it was handed put the process to bed for what may as well be forever. An API retry backoff, a lock-wait loop, a scraper delay — the specific call matters less than the shape: a wait nobody bounded.

Its open file descriptors closed the case:

$ ls -l /proc/793380/fd
0 -> socket:[598208598]
1 -> /usr/local/.../stderr.log (deleted)
2 -> /usr/local/.../stderr.log (deleted)
3 -> socket:[598208598]

Two descriptors on the listening socket. And stdout and stderr pointing at a log file marked (deleted) — something removed that file while the process held it open, so every byte it wrote for fourteen hours went to an inode with no name. If it printed a fatal error, a stack trace, a reason, that is gone. Never let cleanup delete a log a live process still holds open; rotate it and signal the writer instead.

Why nothing restarted it

Here is the part that generalizes past any one server.

A process supervisor decides whether to start a pool by asking two questions: does the socket file exist, and is the PID in the pid file alive?

srwxr-xr-x 1 ... /tmp/.../siteuser.sock
-rw-r--r-- 1 ... /tmp/.../siteuser.sock.pid

Both answers were yes. The socket file was on disk with the right permissions. The pid file named a process that was alive — sleeping, low priority, adopted by init, but very much running.

So the supervisor concluded the pool was up and never spawned a replacement. Meanwhile the orphan, asleep inside a timer, accepted nothing. Every incoming connection landed on a socket with no reader, waited out the connect timeout, and became a 503.

The pool was neither alive nor dead. It was occupying the identity of a working pool — the socket path and the pid — without doing any of the work. A liveness check that asks "is the PID alive" cannot see this, and neither can anything watching for a crash, because nothing crashed.

Fourteen hours

I went back to the access log and counted requests per hour:

That is the moment of death, and it matches the process start time to within a few minutes. From 20:00 onward the only requests left are bots, retries, and the handful of people who tried, got nothing, and left.

Nineteen requests an hour is not zero, and that is the trap. A "traffic dropped to zero" alert never fires. A daily-total alert compares a bad day against a normal one and shrugs at a 90% drop if the threshold is 95%. The site served errors to every visitor for fourteen hours while the graph showed a line that was low but not flat.

The fix, and it is boring

kill -TERM 793380
rm -f /tmp/.../siteuser.sock /tmp/.../siteuser.sock.pid

Kill the orphan, remove the stale socket and pid file so the supervisor stops believing a pool exists, then send one request. The next request rebuilt the pool by itself. Local response: 200 in 0.171 seconds. From outside: 200 in 0.29 seconds, full page, no error signatures in the HTML. Four healthy children within seconds.

Under ten seconds of repair after fourteen hours of downtime. That ratio is the actual story. The failure was trivial to fix and completely invisible.

What to check instead

Monitor the URL users actually load, and follow redirects to the end. A 302 from a server-level rewrite is not evidence that your application runs. If your monitor reports on the first response in the chain, it is monitoring your rewrite rules.

Assert on the body, not the status code. Match a string that only appears when the application really rendered — a title, a footer, a token in the markup. Byte length works too: a 503 page is a few hundred bytes, a real page is tens of kilobytes.

Alert on relative traffic collapse, not on zero. A drop from 306 to 19 requests in one hour is a far better signal than any absolute floor, and it fires within the hour instead of the next morning.

Treat long-lived pool children as suspect. Pool workers are meant to be recycled. Any child running for hours with almost no CPU deserves a look. ps -eo pid,ppid,etime,time,cmd plus /proc/<pid>/wchan costs nothing in a cron job, and a worker with PPID 1 is an orphan by definition.

Cap every sleep and every outbound call. The root cause under all of this is an unbounded wait. Timeouts on HTTP clients, bounded retry backoffs, lock waits that eventually give up. A worker allowed to sleep forever will eventually do exactly that.

Never delete a log file a process still holds open. Rotate and signal. Otherwise the one artifact that would have told you why is writing into an unlinked inode.

The short version

DNS, TCP, TLS and the certificate can all be perfect while the site returns 503 to every visitor.
External processor is not available means the web server could not get a working connection to the PHP pool. It is a backend problem, not a network or certificate problem.
A pool child whose parent has died gets PPID 1 and is supervised by nothing.
If that orphan keeps the socket and pid file, the supervisor sees a live pool and refuses to spawn a real one. Neither alive nor dead, just occupying the identity.
/proc/<pid>/wchan naming a sleep function, with hours of elapsed time and seconds of CPU, is the confirmation.
Health checks that follow redirects and assert on body content would have caught this in the first minute.

The site was down for fourteen hours. The fix was one signal and two deleted files. Everything expensive about this outage happened in the gap between a process existing and a process working.

Written from production work at Alesta WEB.

Lowercasing "İ" Adds a Character: How Turkish Text Broke a Keyword Router

Mahmut Gündüzalp — Fri, 04 Sep 2026 06:41:21 +0000

A classifier was sending questions to the wrong bucket. Not occasionally. I measured it against twenty real inputs and seven of them landed somewhere they did not belong. Thirty-five percent.

The routing code was thirty lines long and did the obvious thing: lowercase the incoming text, then check whether any keyword for a category appears in it. First match wins.

Every one of those thirty lines was wrong in a way that only shows up in Turkish, and two of the three problems will bite you in any language whose case mappings are not one-to-one.

The symptom

Users type free-form questions. The router picks a topic so that the right prompt and the right reference data get loaded. When it picks wrong, the answer is confidently about the wrong subject, which is worse than no answer.

The reported complaint was "the answers feel shallow." The instinct is to blame the model. The model was fine. It was being handed the wrong context.

Trap 1: lowercasing a dotted capital I produces two code points

Here is the whole thing, measured rather than remembered:

$s = "İş";                              // "work", capital İ
$l = mb_strtolower($s, "UTF-8");

// original      : U+0130 U+015F
// mb_strtolower : U+0069 U+0307 U+015F
// mb_strlen     : 2 -> 3

U+0130 LATIN CAPITAL LETTER I WITH DOT ABOVE has no single-code-point lowercase form in the default Unicode case mapping. It maps to U+0069 LATIN SMALL LETTER I followed by U+0307 COMBINING DOT ABOVE. The string got longer. It still renders as "i̇ş" and looks entirely normal in your terminal, your editor, and your database client.

So this happens:

mb_strpos(mb_strtolower("İşten çıkarılır mıyım", "UTF-8"), "iş");   // false
mb_strtolower("İSTANBUL", "UTF-8") === "istanbul";                  // false

Your keyword is two code points. The haystack has three. No match, no error, no warning. The comparison returns false and the request goes to whatever category happens to be checked next.

The reason this is not an edge case: people capitalize the first word of a sentence. In Turkish a large number of everyday words begin with İ — the words for work, request, relationship, reputation, tender, transaction, need, cancellation. Any of them starting a sentence produces the three-code-point form. The broken form is the common form, and the version your test uses — all lowercase, typed by you — is the rare one.

That is why this survives a green test suite. You wrote "işten çıkarılır mıyım" in the fixture, in lowercase, by hand. It matches. The user wrote "İşten çıkarılır mıyım". It does not.

Unicode normalization will not save you

The reflex is to reach for Normalizer::normalize() and move on. I checked:

lower    : U+0069 U+0307 U+015F
NFC      : U+0069 U+0307 U+015F
NFKC     : U+0069 U+0307 U+015F
casefold : U+0069 U+0307 U+015F

NFC composes a base character with a combining mark when a precomposed code point exists. There is no precomposed "latin small letter i with dot above" — a lowercase i already has a dot, so Unicode never needed one. The sequence stays two code points under every composing form, and casefold() does not touch it either.

NFD is more interesting, because it goes the other way and decomposes everything:

NFD("i̇ş") : U+0069 U+0307 U+0073 U+0327

Now the cedilla under ş is its own code point too. That gives you one rule that handles both problems at once: decompose, then drop every nonspacing mark.

import unicodedata as u

def norm(s: str) -> str:
    s = u.normalize('NFD', s.lower())
    s = ''.join(c for c in s if u.category(c) != 'Mn')
    return s.replace('ı', 'i')      # dotless ı has no combining mark to strip

Measured output:

'İş'         -> 'is'
'İSTANBUL'   -> 'istanbul'
'Değişiklik' -> 'degisiklik'
'İlişki'     -> 'iliski'
'Ölüm'       -> 'olum'

In PHP, where the intl extension is frequently not loaded, the same thing without a normalizer:

function tr_norm(string $s): string {
    $s = mb_strtolower($s, 'UTF-8');
    $s = str_replace("\u{0307}", '', $s);        // the leftover dot from İ
    return strtr($s, [
        'ı'=>'i','ş'=>'s','ğ'=>'g','ü'=>'u','ö'=>'o','ç'=>'c',
        'â'=>'a','î'=>'i','û'=>'u',
    ]);
}

The single most important line is the U+0307 removal. Delete it and the bug is back.

Run your keyword list through the same function, too. Both sides of a comparison must see the same normalization, always. A keyword table normalized at author time and input normalized at request time will drift the moment somebody edits the table by pasting from a document.

The other direction, for readers who do not write Turkish

Turkish has two distinct letters where English has one: dotted İ/i and dotless I/ı. They are different letters, not styling. So the correct Turkish lowercase of I is ı, not i.

This is the well-known "Turkish I problem," and it usually appears as the mirror image of what I described. In Java, .NET, or any locale-sensitive toLowerCase() running under a tr-TR locale:

"INFO".toLowerCase()          // "ınfo" under tr-TR, "info" everywhere else
"FILE".toLowerCase()          // "fıle"

Code that lowercases a protocol keyword, a header name, a file extension, or an SQL identifier and then compares it against an ASCII literal breaks on a machine whose locale happens to be Turkish. The fix there is the opposite of the one above: pass an explicit invariant locale, toLowerCase(Locale.ROOT), and never let ambient locale decide the semantics of a comparison.

So there are two rules, pointing in opposite directions depending on what you are doing:

Comparing protocol tokens and identifiers: force a locale-invariant mapping. Never fold, never guess.
Comparing human text for search or classification: normalize aggressively, on both sides, with a function you control.

Confusing the two is how you end up with a system that is broken in one country and subtly wrong everywhere else.

Trap 2: unbounded substring matching

With normalization fixed, the router still misrouted. The second bug had nothing to do with Unicode: the check was strpos($text, $keyword) !== false, with no word boundary.

Short keywords swallow longer words that merely contain them. Real collisions from the keyword table:

Keyword	Means	Also matches	Which means
`olum`	death	olumlu, olumsuz	positive, negative
`mal`	goods, property	normal, mal**zeme	normal, material
`is`	work	kisi, degisiklik, gise	person, change, counter
`ask`	love	maske	mask
`din`	religion	aydin, din**len	bright, rest

olum was the expensive one. "Olumlu" — "positive" — is among the most common words in any yes/no question. Every such question was routed to the death-and-inheritance category, because that category happened to sit near the top of the map.

Word boundaries in an agglutinative language are not just \b around a fixed string. Turkish stacks suffixes, and stems mutate: "lost" appears as kayıp, kaybettiğim, kaybolan — the stem itself shifts from kayıp to kayb. So match a stem plus a bounded run of following letters:

$pattern = '/(?<!\p{L})' . preg_quote($stem, '/') . '\p{L}{0,6}(?!\p{L})/u';

(?<!\p{L}) and (?!\p{L}) are Unicode-aware boundary assertions — plain \b reasons about \w, whose meaning depends on flags and locale. The {0,6} allows suffixes without letting the stem float in the middle of an unrelated word. And keep a stem list per concept rather than one string: kayıp|kayb|kaybol|kaybet, because sound changes will not occur to you at the keyboard, only in production.

Trap 3: map order is a priority ranking you did not know you wrote

foreach ($map as $topic => $words) {
    foreach ($words as $w) {
        if (matches($text, $w)) return $topic;   // first hit wins
    }
}

Whatever order the array literal happens to be in is your precedence rule. "Will my father's health improve" contains both a family keyword and a health keyword. Family was declared fourth, health sixth, so it went to the family bucket every time.

Two changes fix the class of problem rather than the instance:

Order specific before general, deliberately, with a comment saying so.
Stop returning on first hit. Collect every match, score them, take the best. A reasonable score is the number of distinct stems matched, weighted by stem specificity — a rare six-letter stem is worth more than a three-letter one that appears everywhere.

Scoring also gives you something first-match never can: a confidence number. When the top two topics score within noise of each other, you can fall back to a neutral path instead of committing to a coin flip.

Normalize for matching, never for identity

One caution, because the recipe above is lossy on purpose:

'ILIK'  -> 'ilik'
'ılık'  -> 'ilik'

Two different words collapse to the same key. That is exactly what you want for search and classification, and exactly what you must never do for identity.

If you normalize usernames or email local parts this way before a uniqueness check or a login comparison, you have created an account-collision path: two visually and semantically distinct identifiers now compare equal. The same applies to case-insensitive uniqueness in general — the database collation, the application-level check and the login comparison have to agree on one rule, and that rule should be locale-invariant, not a fuzzy fold.

Fold for relevance. Compare exactly for identity. Keep the two functions in separate files so nobody reuses one for the other by accident.

How to verify, because you cannot eyeball this

Every symptom in this article is invisible on screen. The three-code-point string looks identical to the two-code-point one. You cannot review your way to correctness here; you have to measure.

What I did, and would do again:

Write a table of 20–30 real inputs with the expected category, taken from actual traffic rather than invented. Run it, print the wrong-rate as a number. "Seven of twenty" is a fact you can act on; "seems better now" is not.
Sweep for collisions mechanically. For every keyword shorter than about five characters, generate common words containing it and feed them through the classifier. That is how olum inside olumlu surfaced — nobody was going to catch it by reading.
Print code points, not strings, whenever a comparison fails for no visible reason. printf("U+%04X ") over the characters takes ten seconds and ends the argument.
Re-run the same table after the fix and report the new number. A fix that is not re-measured is a hypothesis.

The short version

mb_strtolower('İ') returns i + U+0307. The string gets longer, looks the same, and stops matching.
NFC and casefold do not collapse it. NFD plus stripping nonspacing marks does.
Normalize the input and the keyword list with the same function.
Match stems with Unicode-aware boundaries, not bare substrings.
Map order is silent precedence. Score instead of returning on first hit.
Fold for search; compare exactly for identity.
Measure the wrong-rate before and after, on real inputs.

None of this is exotic Unicode trivia. It is one line of case mapping and one missing word boundary, and together they were quietly discarding a third of the routing decisions in a system that looked, from the outside, like a model quality problem.

Written from production work at Alesta WEB.

The Word "null" Is Not a Null: How JSON Nulls Turn Into Text on Android

Mahmut Gündüzalp — Sun, 30 Aug 2026 10:39:10 +0000

A user opened our mobile app and saw this on their profile screen:

Membership: null
Points: null
Last payment: null

The same account on the website showed empty fields and a zero. No crash, no error dialog, no red line in the log. The app had received the data, parsed it successfully, and printed the four characters n, u, l, l into a TextView.

The parsing code looked like it had already handled this:

val plan = obj.optString("membership", "")

That is the whole bug. On Android, when membership is JSON null, optString does not return "". It returns the string "null".

Why it does that

org.json on Android represents JSON null with a sentinel object, JSONObject.NULL. It is not Java null — it is a real object that exists specifically so the library can tell "the key is absent" apart from "the key is present and its value is null."

Android's implementation of optString is, in essence:

public String optString(String name, String fallback) {
    Object object = opt(name);
    String result = JSON.toString(object);   // ~ object.toString()
    return result != null ? result : fallback;
}

opt("membership") returns the sentinel. The sentinel is not Java null, so the fallback branch is never reached. toString() is called on it, and JSONObject.NULL.toString() returns — by design, because it has to serialize back to valid JSON — the text "null".

The fallback works exactly as documented for a missing key. It does nothing for a null key. Those are two different states and the API gives them one parameter.

getString has the same outcome. It does not throw on a null value; it hands you "null" as well.

Now compare the neighbours:

obj.optInt("points", 0)        // JSON null -> 0     ✅
obj.optBoolean("active", false) // JSON null -> false ✅
obj.optString("plan", "")       // JSON null -> "null" ❌

optInt and optBoolean route through converters that return Java null for the sentinel, so the fallback fires. Only the string path stringifies the sentinel. This inconsistency is why the bug survives review: three lines that look identical, and one of them behaves differently.

It also depends on which `org.json` you are compiled against

This is worth checking rather than assuming. The reference implementation from json.org — the org.json:json jar you would pull into a plain JVM or server-side project — has a version of the same method that reads roughly:

public String optString(String key, String defaultValue) {
    Object object = this.opt(key);
    return NULL.equals(object) ? defaultValue : object.toString();
}

It checks the sentinel explicitly and returns your default. Same package, same class, same method signature, different answer.

Android ships its own Harmony-derived implementation in the framework, and it wins on the classpath. So a unit test that runs on the JVM with the json.org jar can pass while the device fails. If your test source set is green and your androidTest source set is red on the same assertion, this is a candidate.

Confirm it in ten seconds with an instrumented test rather than trusting either the docs or this article:

@Test fun optStringOnJsonNull() {
    val o = JSONObject("""{"a": null}""")
    assertEquals("", o.optString("a", ""))   // fails on device
}

Where the null came from in the first place

Fixing the client is only half of it, because the response should not have contained nulls for those fields at all.

The API was PHP, and the payload was built from a query that looked something like:

SELECT
    m.plan_name,
    SUM(p.amount)  AS total_paid,
    MAX(p.paid_at) AS last_payment
FROM members m
LEFT JOIN payments p ON p.member_id = m.id
WHERE m.id = ?
GROUP BY m.id

Two separate null factories in one query:

LEFT JOIN with no matching rows. Every column from payments comes back NULL.
Aggregates over an empty set. SUM() and MAX() return NULL, not 0 and not '', when there are no rows to aggregate. COUNT() is the exception — it returns 0 — which is exactly why people assume the others do too.

The website never showed this because the view layer cast on the way out:

echo (int)($row['total_paid'] ?? 0);
echo htmlspecialchars($row['plan_name'] ?? '');

The ?? 0 there is doing real work — ?? catches SQL NULL once PDO has turned it into PHP null. Web output was clean. The JSON endpoint serialized the same row directly, json_encode mapped PHP null to JSON null, and the app printed the word.

This is the general shape: a null that is invisible in one rendering path becomes visible in another. Two consumers of the same data, one of which happens to have a defensive cast in its template.

Why the test accounts never caught it

The seeded test users all had payment history. Every demo account had rows. The only way to produce this response was to log in as a member with nothing — a brand new registration, or an account whose records had been archived.

A user with zero rows is a boundary condition, and it is the one that seed scripts systematically do not create, because seed scripts exist to make screens look populated. If you write fixtures, make one account deliberately empty and put it in the smoke test.

Fix both layers, not one

The tempting move is to fix the server, since that is where the null originates, and declare the client fine. That is wrong for mobile specifically, and here is why.

Old app versions live forever. You control your server. You do not control which build is on a user's phone. A client that renders "null" when handed a JSON null is a permanently loaded gun pointed at every future endpoint, including ones written by someone who has never heard of this bug.

Caches outlive deploys. An HTTP cache, a CDN, or a local response cache can hand the app a pre-fix payload well after the server is patched.

New fields arrive constantly. Server-side normalization is a policy you have to remember to apply. Client-side hardening is a helper you cannot forget to use, because it is the only way to read a string.

So: both.

Server side — normalize once, at the boundary

Recursively convert nulls at the point where the response is serialized, not in each handler:

function json_clean(mixed $value): mixed
{
    if (is_array($value)) {
        foreach ($value as $k => $v) {
            $value[$k] = json_clean($v);
        }
        return $value;
    }
    return $value ?? '';
}

// single exit point for every API response
function api_respond(array $payload, int $status = 200): void
{
    http_response_code($status);
    header('Content-Type: application/json; charset=utf-8');
    echo json_encode(json_clean($payload), JSON_UNESCAPED_UNICODE);
}

One caveat worth stating plainly: this flattens the difference between "no value" and "empty value". For a display API that is what you want. For an API where a client needs to distinguish "field not provided" from "field explicitly cleared" — a PATCH-style contract, for instance — do not do this globally. Fix those fields at the query instead:

COALESCE(SUM(p.amount), 0)   AS total_paid,
COALESCE(m.plan_name, '')    AS plan_name

COALESCE is the honest fix; the recursive normalizer is the safety net under it.

Client side — never call `optString` directly again

Wrap it once, ban the raw call in review:

object Json {

    fun text(obj: JSONObject?, key: String, fallback: String = ""): String {
        if (obj == null || !obj.has(key) || obj.isNull(key)) return fallback
        val value = obj.optString(key, fallback).trim()
        // a server can also send the *string* "null" — from string concatenation,
        // a logging wrapper, or an older client that echoed this bug back
        if (value.isEmpty() || value == "null" || value == "undefined") return fallback
        return value
    }

    fun number(obj: JSONObject?, key: String, fallback: Long = 0): Long {
        if (obj == null || obj.isNull(key)) return fallback
        return obj.optLong(key, fallback)
    }

    fun textOrNull(obj: JSONObject?, key: String): String? =
        text(obj, key, "").ifEmpty { null }
}

isNull(key) is the load-bearing line. It returns true for both Java null and the JSONObject.NULL sentinel, which is the check optString fails to make on your behalf.

Filtering the literal "null" string looks paranoid until you have seen a server do "Welcome, " + name in a template and store the result. Once a "null" has been written to a database by an upstream system, no amount of correctness downstream removes it — you have to strip it at read time.

Then make the raw call unusable:

// build.gradle.kts — fail the build, not the user
// (or use Detekt's ForbiddenMethodCall rule)

A lint rule beats a code review comment, because the review comment only catches the calls written while the reviewer remembers the bug.

What about Gson, Moshi, kotlinx.serialization?

Mostly they handle this correctly, with one trap of their own.

Moshi and kotlinx.serialization are aware of Kotlin nullability. Declare val plan: String as non-null and feed it JSON null, and they throw at parse time — a loud, immediate, correctly-located failure. That is the behaviour you want.

Gson does not. It constructs objects through unsafe reflection and bypasses Kotlin's null checks entirely, so a JSON null lands in a field the type system swears is non-null. Nothing fails at parse time. The NPE happens later, somewhere unrelated, and the stack trace points at the innocent code that touched the field rather than the parser that lied about it. That is arguably worse than "null" on screen, because at least "null" on screen tells you where it came from.

If you are on Gson, declare the fields nullable (String?) and handle the nulls explicitly. Fighting for non-null fields you cannot enforce buys nothing.

org.json remains everywhere despite all three of these, for a simple reason: it is in the Android framework, weighs nothing, and needs no annotation processor. Small apps and old modules keep using it. It works fine — as long as you know that its string path does not have a null story.

The five-minute audit

Worth running on any app that talks to an API you also control:

grep -rn "optString(" app/src — every hit is a candidate. Replace with a helper that calls isNull first.
grep -rn "getString(" app/src on JSONObject receivers — same problem, no fallback parameter to give false comfort.
In the API, find aggregates without COALESCE: SUM(, MAX(, MIN(, AVG( — and every LEFT JOIN selecting columns from the right-hand table.
Create one test account with zero related rows and open every screen with it. This finds more null bugs in ten minutes than reading code for a day.
grep your database for the literal string 'null' in text columns. If you find any, something upstream has already been concatenating nulls into text, and that data needs cleaning independently of any code fix.

The underlying lesson is not about one library method. It is that null crosses system boundaries badly. SQL NULL, PHP null, JSON null, Java null, Kotlin null?, and a sentinel object that stringifies to "null" are six different things wearing the same name, and every hop between two of them is a place where one quietly becomes another. Put the conversion in one function per boundary, and you only have to be right once.

I build and run news, e-commerce and membership platforms at alestaweb.com. This one took an afternoon and a screenshot from a user who assumed "null" was our membership tier.

MySQL Will Quietly Eat Your Data If You Let It: Truncation, Charsets, and the 1366 You Never See

Mahmut Gündüzalp — Sun, 23 Aug 2026 08:30:21 +0000

A WhatsApp button on a shop was dialling the wrong number. Not a formatting problem — a genuinely different number, three digits short of the one in the admin panel.

The admin panel had saved it. The page was reading it. Nothing errored. The column was varchar(10), the number arrived with a country code, and MySQL had done exactly what it was configured to do: chop the value to fit and carry on.

That column had been wrong for months. Nobody noticed because nothing anywhere in the stack raised its voice.

Here is the family of bugs that behaves this way, how to find them in a database you inherited, and why the obvious fix — "just turn on strict mode" — creates a different bug if you do only that.

Part 1: The truncation

Run this on any MySQL or MariaDB instance:

SELECT @@GLOBAL.sql_mode, @@SESSION.sql_mode;

If neither contains STRICT_TRANS_TABLES or STRICT_ALL_TABLES, then for every INSERT and UPDATE your server is in the business of making data fit, not of telling you it doesn't.

What that means concretely:

CREATE TABLE t (phone VARCHAR(10));
INSERT INTO t VALUES ('905551234567');
SHOW WARNINGS;
-- Level: Warning  Code: 1265  Message: Data truncated for column 'phone' at row 1

SELECT phone FROM t;
-- 9055512345

Note the level. Warning, not error. The statement succeeded. affected_rows is 1. PDO in exception mode throws nothing, because there is nothing to throw — MySQL considers this a completed statement. Your ORM reports success. Your integration test asserting "the row exists" passes. The value is wrong.

The same mechanism applies well beyond strings:

You wrote	Column	Non-strict result
`'905551234567'`	`VARCHAR(10)`	`'9055512345'`
`300`	`TINYINT`	`127`
`'2026-02-31'`	`DATE`	`'0000-00-00'`
`''`	`INT NOT NULL`	`0`
`12.999`	`DECIMAL(4,2)`	`13.00`

Every one of those is a silent, permanent difference between what your application believed and what your database holds. And unlike a crash, there is no timestamp to correlate against — you cannot tell from the row when it happened or how many rows before it went the same way.

Finding the damage in a database you inherited

You cannot recover truncated values — the tail is gone. But you can find the columns where it is happening, which is what matters going forward. Truncated values pile up at exactly the column limit, so the suspects are the columns where real rows sit at the maximum length:

-- 1. list the string columns and their limits
SELECT TABLE_NAME, COLUMN_NAME, CHARACTER_MAXIMUM_LENGTH AS max_len
FROM information_schema.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
  AND DATA_TYPE IN ('varchar', 'char')
  AND CHARACTER_MAXIMUM_LENGTH <= 64      -- short columns are where this bites
ORDER BY CHARACTER_MAXIMUM_LENGTH;

Then, per interesting column:

SELECT COUNT(*) AS at_the_limit
FROM settings
WHERE CHAR_LENGTH(phone) = 10;

A handful of rows sitting exactly at the limit can be a coincidence. A majority of rows sitting exactly at the limit is not a coincidence, it is a report. In our case the count was every row that had ever been saved with a country code.

Two details that matter when you run this:

Use CHAR_LENGTH(), not LENGTH(). LENGTH() counts bytes; CHAR_LENGTH() counts characters. On multi-byte text they disagree, and VARCHAR(n) limits characters, not bytes.
Do this on a replica or a restored dump if the table is large. CHAR_LENGTH() in a WHERE clause means a full scan.

Part 2: Strict mode is necessary and not sufficient

The obvious response is to turn strict mode on. Do it — but understand what you are buying.

Strict mode does not repair the mismatch between what your application sends and what the column accepts. It changes how you find out about it: instead of quietly storing a wrong value, MySQL raises error 1406 (Data too long for column) and your code, which never expected a write to fail, hands the user a 500.

I have seen exactly this on an admin form. Strict mode was correctly enabled; the title field in the form had no maxlength and no server-side length check; the column was varchar(150). Someone pasted a long headline. Instead of a validation message the editor got a blank error page, and — worse — a story they thought they had saved and hadn't.

That is still better than silent corruption, because it is loud and it is immediate. But the actual fix is at the boundary:

$rules = [
    'title' => ['max' => 150],   // same number as the column, in one place
    'phone' => ['max' => 20],
];

foreach ($rules as $field => $rule) {
    if (mb_strlen($input[$field] ?? '') > $rule['max']) {
        return back()->withError("{$field}: at most {$rule['max']} characters.");
    }
}

Three habits that keep this honest:

Validate before you write, with the column's real limit. Generate the limits from information_schema if you can — a hand-copied number drifts the first time someone runs an ALTER.
Turn strict mode on so the boundary check has a backstop. Validation you wrote can be bypassed; the database is the last honest party in the chain.
Widen the column when the data is legitimately bigger. VARCHAR(10) for a phone number was never right. Truncation is a symptom; a wrong schema is the disease.

Part 3: Charsets, where "it worked on the old server" lives

A database was copied to a new server. Everything imported. Row counts matched. A week later somebody noticed that Turkish characters in older records had turned into question marks in some tables and not others.

The mistake was trusting the database default:

SHOW CREATE DATABASE app;
-- CHARACTER SET utf8mb4  ← looks fine, means almost nothing

The database default applies to newly created tables that do not specify their own. It says nothing about the columns you already have. The columns are where the truth is:

SELECT TABLE_NAME, COLUMN_NAME, CHARACTER_SET_NAME, COLLATION_NAME
FROM information_schema.COLUMNS
WHERE TABLE_SCHEMA = 'app'
  AND CHARACTER_SET_NAME IS NOT NULL
  AND CHARACTER_SET_NAME <> 'utf8mb4';

If that returns rows, you have a mixed-charset database, and every dump/restore across it is a chance to lose bytes.

Two more charset facts worth having in your head:

utf8 in MySQL is not UTF-8. The alias utf8 (historically utf8mb3) stores at most three bytes per character, so it cannot hold anything outside the Basic Multilingual Plane — emoji, some CJK extensions, some historic scripts. Insert one into a utf8 column and, in non-strict mode, MySQL truncates the value at the emoji and keeps whatever came before. utf8mb4 is real UTF-8. Use it everywhere, including the connection charset.

The connection charset is part of the pipeline. Your column can be utf8mb4, your data can be perfect, and your terminal or client can still show you mojibake because the session negotiated something else. Which leads to the most useful debugging habit in this whole article:

When you think data is corrupted, check the bytes

SELECT title, HEX(title), LENGTH(title), CHAR_LENGTH(title)
FROM articles WHERE id = 42;

HEX() is the ground truth. It is not affected by your client, your terminal font, or the connection charset. I have twice now "found" data loss that turned out to be a display artefact of the command-line client, and both times HEX() settled it in one query: the bytes were intact, the rendering was not.

The tell for genuine UTF-8 content is LENGTH() > CHAR_LENGTH(). If they are equal on text you know contains non-ASCII characters, something upstream already flattened it.

Part 4: The application-side version of the same bug

Databases are not the only layer that cuts strings. This is a summary excerpt in PHP:

$summary = substr($body, 0, 200);   // wrong

substr() counts bytes. On UTF-8 text, byte 200 lands in the middle of a multi-byte character about half the time, and you send MySQL a string ending in half a character. Then either:

the column is utf8mb4 and strict mode is on → error 1366, Incorrect string value, and you get a 500 from what looked like a formatting line; or
strict mode is off → MySQL drops the invalid tail silently, and your summaries are quietly one character shorter than you think, sometimes ending in a replacement glyph.

The fix is one letter and three characters:

$summary = mb_substr($body, 0, 200);   // counts characters

Whenever you see strlen, substr, strtoupper, or str_pad applied to user text, treat it as a bug report waiting to happen. Error 1366 in a log almost always traces back to one of them.

The checklist

Run these on any project you did not set up yourself. They take about ten minutes together.

SELECT @@GLOBAL.sql_mode; — is STRICT_TRANS_TABLES present? If not, plan to enable it after auditing lengths, not before.
List varchar columns with small limits, then count rows sitting exactly at the limit. That count is your truncation report.
List columns whose CHARACTER_SET_NAME is not utf8mb4. Fix them before the next migration, not during it.
Confirm the connection charset your application actually negotiates — the DSN, not the config file you hope it reads.
Grep the codebase for byte-based string functions applied to user content, and replace them with the mb_ versions.
When you suspect corruption, run HEX() before you conclude anything. Half the time the data is fine and the client is lying.

The theme across all of this: the database is willing to accept an approximation of what you gave it, and by default it will not argue. Make it argue. Then make sure your code is ready to hear it.

I build and run news and e-commerce platforms at alestaweb.com. Every example above cost somebody a real afternoon.

Your Uptime Monitor Says 200. Your Contact Form Has Been Dead for Three Days.

Mahmut Gündüzalp — Sun, 23 Aug 2026 08:27:02 +0000

The contact form on a company site I look after stopped producing leads. Not "fewer leads" — zero, for three days. No alert fired. The uptime monitor was green the entire time. The status page had a nice unbroken bar.

The monitor was checking the homepage. The homepage was fine. /contact had been returning a fatal error since a Tuesday afternoon deploy.

This is the most boring class of outage there is, and it is the one that costs the most, because nothing tells you it is happening. Below is what actually broke, the four ways a site can be broken while still answering 200, and the small set of checks I run now instead of pinging the root URL.

What broke

The site loads a list of helper files at boot. During an unrelated edit, that list was rewritten, and one entry did not survive the rewrite — a helper that supplied data to a few templates.

Most pages never touch that helper, so they kept rendering. One page called it in a template. That page fataled. In production, with display errors off, a fatal means an empty response body or a generic error page — depending on how the stack is configured, sometimes with a 500 status, sometimes with the web server's own branded page and a status you did not choose.

The error log had the answer on line one, at the exact timestamp of the deploy. Nobody looked, because nothing said to look.

Three days of a form that a campaign was actively driving traffic to. The traffic arrived. The page did not.

Four ways to be broken and still return 200

If your health check is curl -o /dev/null -w "%{http_code}" https://example.com/ you are testing one route, one time, for one property. Here are the failure modes that sail straight through it.

1. The route you don't check. A site is not a URL, it is a few dozen to a few hundred routes. Homepages are the least likely page to break, because everything touches them and everyone looks at them. The pages that break are the ones with one unusual dependency: the form page, the search page, the report that joins four tables.

2. The page renders, but a part of it silently didn't. Templates are forgiving by design. A partial that throws inside a try/catch, a widget whose data source returned an empty array, a block that renders only when a config key exists — none of these change the status code. The page looks right at a glance and is missing the thing that made it worth serving.

I found one of these in the same audit: a function that emitted structured data (Article, BreadcrumbList) existed, was correct, and was called from nowhere. Thirty news URLs had been building JSON-LD that never reached the HTML. Status: 200. Value delivered: none.

3. The page is fine and the form is not. This one is hard to see. A full-page HTML cache was turned on. HTML caches are dumb by nature: they store bytes. Those bytes included a CSRF token. Every visitor got the first visitor's token, every POST failed validation, and the failure looked to the user like a page reload with no message. Every page on the site: 200. Every form on the site: broken.

4. The error page returns 200. Custom error handlers that render a friendly "something went wrong" view and forget to set the status. Frameworks that catch late and fall back to a template. Reverse proxies serving a static maintenance page with the default status. Your monitor sees 200 and a body of some length, and reports health.

What to assert instead

The fix is not a bigger monitoring product. It is asserting the things you actually care about. Four checks, in increasing order of value.

Check every route, not one route

You already have the route list — it is in your router, your sitemap, or both. Walk it.

#!/usr/bin/env bash
# routes.txt: one path per line, from the sitemap or the router
FAIL=0
while read -r path; do
  code=$(curl -s -o /tmp/body -w '%{http_code}' "https://example.com${path}")
  size=$(wc -c < /tmp/body)
  if [ "$code" != "200" ] || [ "$size" -lt 2000 ]; then
    echo "FAIL ${path} status=${code} bytes=${size}"
    FAIL=1
  fi
done < routes.txt
exit $FAIL

The size floor is doing real work here. A fatal that returns 200 with an empty or near-empty body is caught by bytes, not by status. Pick the floor from your own smallest legitimate page, then subtract a little.

Assert content, not just bytes

One string per route, chosen to be the thing that page exists to do. The form page must contain a <form and a submit button. The article page must contain an <h1. The page with structured data must contain application/ld+json.

grep -q '<form' /tmp/body || echo "FAIL ${path}: no form in body"
grep -qi 'fatal error\|<b>Warning</b>\|stack trace\|Undefined variable' /tmp/body \
  && echo "FAIL ${path}: error signature in HTML"

That second grep catches the inverse case: the page that renders and prints a warning into the output where a visitor — or a crawler — can read it.

Diff the error log around every deploy

This is the cheapest high-value check in the list and almost nobody does it. Record the log size before the deploy, read the tail after.

BEFORE=$(stat -c%s /path/to/error.log)
# ... deploy ...
sleep 30
AFTER=$(stat -c%s /path/to/error.log)
if [ "$AFTER" -gt "$BEFORE" ]; then
  tail -c $((AFTER - BEFORE)) /path/to/error.log
  echo "New errors appeared during deploy — investigate before walking away."
fi

In our incident this would have printed the missing helper, by name, thirty seconds after the deploy that caused it. The information was there the whole time. Nothing surfaced it.

Round-trip the form

The only way to know a form works is to submit it. A synthetic submission is about twenty lines and can run hourly:

<?php
// 1. Fetch the page like a browser would, keeping cookies.
$ch = curl_init('https://example.com/contact');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_COOKIEJAR      => '/tmp/probe-cookies',
    CURLOPT_COOKIEFILE     => '/tmp/probe-cookies',
]);
$html = curl_exec($ch);

// 2. Pull the CSRF token out of the markup, exactly like a browser.
preg_match('/name="_token" value="([^"]+)"/', $html, $m);
$token = $m[1] ?? null;
if (!$token) { exit("FAIL: no CSRF token on the page\n"); }

// 3. Submit with a marker you can find and delete later.
$marker = 'probe-' . date('YmdHi');
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query([
    '_token'  => $token,
    'name'    => $marker,
    'email'   => 'probe@example.com',
    'message' => 'automated probe, safe to delete',
]));
curl_exec($ch);

// 4. Assert the effect, not the response.
$found = (int) $pdo->query(
    "SELECT COUNT(*) FROM leads WHERE name = " . $pdo->quote($marker)
)->fetchColumn();

echo $found === 1 ? "OK\n" : "FAIL: submitted, nothing stored\n";

Step 4 is the point of the whole thing. Not "did the POST return 200" — did a row appear. That single assertion catches the frozen-token case, the silently failing mail transport, the validation rule someone tightened, and the database column that quietly truncated the input.

Give the probe an obvious marker and delete its rows on a schedule. Yes, this writes to production. That is the trade: a handful of tagged test rows a day against not knowing your lead form is dead.

Why this keeps happening

Three reasons, and none of them are laziness.

Monitoring defaults are shaped by hosting, not by the product. A default check answers "is the machine up and serving". That was the right question when a machine going down was the common failure. The common failure now is code that is up and wrong.

Green is a feeling, and feelings are expensive to give up. An unbroken status bar makes people stop looking. Silence should mean "the assertions passed". In practice it means "nothing asserted anything".

Silent failure has no symptom by definition. A crash gets escalated in an hour. A form that swallows submissions gets noticed when someone asks why the pipeline is empty — which, in our case, took three days, and only because someone went looking for a different number.

The short version

A status code is a transport fact. It says the request reached something. It does not say the page did its job.
Check every route, not the homepage. Your sitemap already lists them.
Assert one meaningful string per route, and grep the body for error signatures while you are there.
Diff the error log across every deploy, automatically. Cheapest check here.
Round-trip the forms that make you money, and assert the stored row, not the response.
Anything that renders "sometimes" — structured data, widgets, conditional blocks — needs an output assertion, because code review will not catch a function that is never called.

None of this is sophisticated. The whole set is a cron job and about a hundred lines. The reason to write it is that the alternative is finding out from a person, and by then you have already paid for the traffic.

I build and maintain news and e-commerce platforms at alestaweb.com. Most of what I know about monitoring came from outages exactly this dumb.

The Query That Ate 75% of Our Database CPU: A MySQL Full-Text Post-Mortem

Mahmut Gündüzalp — Thu, 06 Aug 2026 22:11:21 +0000

A news platform I maintain started serving pages in six seconds. Load average sat at 16 on a 12-core box for hours. Nothing had been deployed. Traffic was up, but not 10x up.

The cause turned out to be a single SELECT that looked completely reasonable — the kind of query that passes code review, works fine on a 5,000-row table, and quietly becomes a wrecking ball at 80,000 rows.

Here is the whole investigation: how I found it, why it was slow, what the fix was, and the three unrelated things I learned along the way.

Symptom first

The obvious metrics:

Load average 16.7 on 12 cores
MySQL at 92% CPU, resident memory 9.8 GB out of 15 GB
4.8 GB pushed into swap
Time to first byte on article pages: 6.3 seconds

The tempting move here is to start tuning. Bump the buffer pool, add more cache, blame the bots. I've done that before and it's mostly a way of not finding the bug.

Find the query, not a query

The mistake I see most often at this stage is taking one snapshot of the process list, spotting something slow, and declaring victory. One snapshot tells you what was running at one instant. It doesn't tell you what dominates.

Sample it instead:

for i in $(seq 1 20); do
  mysql -N -e "SELECT info FROM information_schema.processlist
               WHERE command='Execute' AND info IS NOT NULL"
done | sed -E 's/.*MATCH.*/RELATED-ARTICLES/' | sort | uniq -c | sort -rn

Twenty cheap samples, bucketed by shape. The result:

    202 RELATED-ARTICLES
     41 other article queries
     21 rate-limit counter

202 of 268 active queries — 75% — were the same statement. That's not a slow query problem, that's a "one feature is eating the server" problem.

The query

It powers the "related articles" block under every story:

SELECT a.id, a.title, a.summary, a.image, a.published_at,
       MATCH(a.title, a.summary) AGAINST(? IN NATURAL LANGUAGE MODE) AS relevance
FROM articles a
LEFT JOIN categories c ON a.category_id = c.id
WHERE MATCH(a.title, a.summary) AGAINST(? IN NATURAL LANGUAGE MODE)
  AND a.id != ? AND a.status = '1'
ORDER BY (relevance / (DATEDIFF(NOW(), COALESCE(a.sort_date, a.published_at)) / 30 + 1)) DESC
LIMIT 6

Read that ORDER BY again. It divides the relevance score by the article's age in months, so a strong match from 2019 loses to a decent match from last week. It's a genuinely nice ranking idea. Freshness matters in news.

And the search term passed in? The current article's entire title plus the first 200 characters of its summary.

Why it's expensive

Two things compound.

Natural language mode is not a filter, it's a scorer. Feed it a long string and it matches on any meaningful token in that string. I measured it:

SELECT COUNT(*) FROM articles
WHERE MATCH(title, summary) AGAINST('<full title + 200 chars of summary>'
                                    IN NATURAL LANGUAGE MODE);
-- 16,141

16,141 rows matched out of 80,836 — about 20% of the table. A long natural-language term is a very wide net. Every extra common word widens it further.

The ORDER BY cannot use an index. It sorts on an expression computed per row. The optimizer has no choice:

+----------+----------------+----------------------------------------------+
| type     | key            | Extra                                        |
+----------+----------------+----------------------------------------------+
| fulltext | ft_title_summ  | Using where; Using temporary; Using filesort  |
+----------+----------------+----------------------------------------------+

Using temporary; Using filesort on a fulltext scan is the whole story. To return 6 rows, the server:

scores ~16,000 rows,
materialises them into a temporary table,
computes relevance / (age/30 + 1) for every one of them,
sorts all 16,000,
throws away 15,994.

Measured cost: ~0.22 s of CPU per call. On every article page view. With half the traffic coming from crawlers walking the archive, each hitting a different article — so per-article caching had a near-zero hit rate for exactly the traffic causing the load.

At a handful of article views per second, that one query alone wants more cores than the machine has.

Fix 1: stop searching with a paragraph

The search term should be the subject of the article, not the article. I extract meaningful words from the title only — drop stopwords, drop anything under three characters, cap at eight words:

function searchTermFromTitle(string $title): string
{
    $stop = ['and','the','with','for','from','that','this','was','are','has'];
    $clean = preg_replace('/[^\p{L}\p{N}\s]+/u', ' ', $title);
    $clean = trim(preg_replace('/\s+/u', ' ', $clean));

    $picked = [];
    foreach (explode(' ', $clean) as $word) {
        $lower = mb_strtolower($word, 'UTF-8');
        if (mb_strlen($lower, 'UTF-8') < 3 || in_array($lower, $stop, true)) {
            continue;
        }
        $picked[] = $word;
        if (count($picked) >= 8) break;
    }
    return $picked ? implode(' ', $picked) : $clean;
}

Match count dropped from ~16,000–58,000 (it varied wildly by article) to ~2,500–13,000. Better, not solved.

A note for non-English text: build the stopword list for the language you actually store. Mechanically reusing an English list on Turkish, German or Finnish content will either strip nothing or strip the wrong things, and morphology means a naive list misses inflected forms.

Fix 2: rank in two stages

The real problem isn't the match count, it's sorting all matches by an expression. So don't. Take the top N by raw relevance — which the fulltext index can drive with a bounded sort — then apply the freshness weighting to those N in an outer query:

SELECT t.id, t.title, t.summary, t.image, t.published_at, t.relevance
FROM (
    SELECT a.id, a.title, a.summary, a.image, a.published_at,
           MATCH(a.title, a.summary) AGAINST(? IN NATURAL LANGUAGE MODE) AS relevance,
           COALESCE(a.sort_date, a.published_at) AS rank_date
    FROM articles a
    LEFT JOIN categories c ON a.category_id = c.id
    WHERE MATCH(a.title, a.summary) AGAINST(? IN NATURAL LANGUAGE MODE)
      AND a.id != ? AND a.status = '1'
    ORDER BY relevance DESC
    LIMIT 60
) t
ORDER BY (t.relevance / (DATEDIFF(NOW(), t.rank_date) / 30 + 1)) DESC
LIMIT 6

The expensive sort now runs over 60 rows instead of 16,000. The derived table has a LIMIT, so it can't be merged away — it's materialised, which is precisely what I want here.

Ranking quality barely moves. A result that wins after freshness weighting was already a strong relevance match; it was never sitting at position 4,000.

Results

Measured against real rows, old query vs new, five articles:

	Before	After
5 queries, total	2.571 s	0.572 s
Share of active queries	75%	16%
Load average	16.7	7.4
TTFB, worst page	6.34 s	1.91 s

4.5x on the query. And no caching layer involved — this is the same work, arranged so the database isn't asked to sort a haystack to hand back six needles.

Three things I learned that weren't the bug

InnoDB does not give space back. I pruned about 2.9 million rows from some log tables. data_free went up by 270 MB and the files didn't shrink. Deleted rows leave free pages inside the tablespace, reusable by that table but not returned to the OS. OPTIMIZE TABLE rebuilds it. If your cleanup job only deletes, your disk usage will never reflect it.

A dynamic variable can fail silently. innodb_buffer_pool_size is dynamic on modern MariaDB, so I resized it live. No error. The value didn't change. The reason: innodb_buffer_pool_chunk_size read as 0, and resizing works in chunk units. Always read the variable back after you set it — don't trust the absence of an error.

Long-lived MySQL processes bloat. That instance had been up 85 days with a 1 GB buffer pool and was holding 9.8 GB resident, 4.8 GB of it swapped. After a restart with a properly sized pool: 1.96 GB resident, swap essentially empty, and 8 GB handed back to the OS. Per-connection buffers and temp-table churn accumulate; a restart is sometimes the honest fix.

What I'd take away

Sample the process list, don't snapshot it. The dominant query and the slowest query are usually different queries, and the dominant one is what's hurting you.
Using temporary; Using filesort next to a LIMIT 6 is a smell. It means the server built the whole set to return a fraction of it.
Natural language full-text search scales with term length. A search term built from a whole paragraph is a query against a fifth of your table.
Fix the shape before you add cache. Caching this would have hidden it from users while the crawler traffic — cache-missing by construction — kept the CPU pinned.

The bug had been there for a long time. It only became visible when the archive got big enough and the crawlers got busy enough for those two curves to cross.

I build news and e-commerce platforms in PHP at Alesta WEB, an independent software company running since 2005.

DEV Community: Mahmut Gündüzalp

PHP2Shell (CVE-2026-109618 / CVE-2026-109619): A Practical Patch, Mitigation and Incident Checklist

What the two bugs are

Patch matrix

Which SAPI is exposed

Updating, per platform

Debian

Ubuntu

Third-party Debian/Ubuntu repositories

RHEL, AlmaLinux, Rocky

Remi repository

cPanel (EasyApache 4)

Plesk

DirectAdmin (CustomBuild)

LiteSpeed lsphp packages

Docker official images

Windows and macOS

FrankenPHP, long-running workers, queues

Restart and verify

Temporary mitigations and their limits

Was I hit? A checklist

Still on PHP 8.1 or older?

Two scams riding on this

Exploitation status (as of 22 October)

Sources

My photo.jpg Was a 29-Byte HTML Page: Validating Downloaded Images Before They Reach Production

What went wrong: an ID that looks like a file name

Why curl said everything was fine

A small validator we now run on every image

Then check the live URL, not the local file

Takeaways

Instant ADD COLUMN, Until a FULLTEXT Index Shows Up: A MariaDB Migration That Froze Writes

The setup

Reproducing it

Why FULLTEXT changes the rules

Mistake 1: one statement per column

Mistake 2: letting the server pick the algorithm

What to do when the rebuild is unavoidable

A pre-flight check you can add to your release script

Takeaways

Green Build, Broken Site: Three Next.js 16 Failures You Only See in Production

1. The image library that compiled to nothing

Symptom

Why it only happened in production

Fix

The check that would have caught it

2. Uploads that 404 until the next restart

Symptom

Why

Fix

The check that would have caught it

3. Scheduled posts that published but never appeared

Symptom

Why

Fix

The check that would have caught it

What the three have in common

Two Proxies, One Comma, Every Page a 500

The shape of the deployment

Two header lines are one string

Where it breaks

The obvious fix, and the second failure

What actually fixed it

The secureCookie line is not optional

The neighbouring headers have the same problem

X-Forwarded-For: the last value is the CDN

Canonical redirects: read the first value, or the CDN's own signal

Checklist for "works on staging, 500 behind the CDN"

The general shape

Six Rows Deleted. Three Decrements Applied. No Error.

The query

The measurement

Why nobody catches it in testing

The fixes that don't work

The fix

Finding the damage you already have

The general shape

The Socket Was Open. The Site Was Dead for Fourteen Hours.

What the checks reported

The log line that mattered

The `secureCookie` line is not optional

`X-Forwarded-For`: the last value is the CDN

It also depends on which `org.json` you are compiled against

Client side — never call `optString` directly again