DEV Community: Mahmut Gündüzalp

The Socket Was Open. The Site Was Dead for Fourteen Hours.

Mahmut Gündüzalp — Mon, 07 Sep 2026 06:59:17 +0000

A site had been down since 19:54 the previous evening. Nobody reported it. The monitoring did not fire. When I finally looked, the cause was one process that had outlived its parent and was holding a socket nobody could use.

Every individual check passed. DNS resolved. Port 443 accepted connections. The TLS handshake completed against a valid certificate with two months left on it. The web server was running and serving other sites on the same machine. And every request to that one site returned 503.

The failure lived in the gap between "the process exists" and "the process works," which is a gap most health checks do not cover.

What the checks reported

I started from the outside, because the outside is cheap to measure:

Layer	Check	Result
DNS	resolves to the expected address	fine
TCP	ports 80 and 443 accept	fine
TLS	handshake, chain, expiry	fine, 68 days left
HTTP (port 80)	`GET /`	302 to the HTTPS URL
HTTPS	`GET /`	first a 20-second hang, then 503
Web server	running, other sites on the box served fine	fine

Two of those rows are the reason nobody noticed.

The port-80 row returns 302 because the HTTP-to-HTTPS redirect is a server-level rewrite. It never reaches PHP. A monitor watching http:// sees a chain that starts with a perfectly healthy 302, and if it treats a redirect as up, it reports up. Forever. The application behind it can be completely dead.

The TLS row is worse, because it looks like the most meaningful check in the list. A successful handshake proves the listener is alive and the certificate is valid. It proves nothing whatsoever about whether anything can generate a response body.

The first HTTPS request I made hung for twenty seconds and timed out. The second returned 503 in 143 milliseconds. That inconsistency is itself a clue: the server was trying to hand the request to a backend, waiting, giving up, and eventually not bothering to wait at all.

The log line that mattered

The site's own error log was useless. Its last entry was seventeen days old, because the log level was WARN and this condition never produced a warning at the virtual-host level.

The server-wide log had it, at INFO:

[INFO] [...#site] Connection idle time too long: 301 while in state: 6 ...
[INFO] [...#site] HttpExtConnector state: 0, request body sent: 0,
                  response body size: -2, response body sent: 0, attempts: 0.
[INFO] [...#site] External processor is not available.
[NOTICE] [...#site] oops! 503 Service Unavailable

External processor is not available is the whole diagnosis in five words. The web server had a request, knew which backend should handle it, and could not get a usable connection to that backend. In LSAPI terms the external processor is the PHP process pool. In PHP-FPM terms it is the pool behind the socket you point fastcgi_pass at. Same shape, same failure mode.

Note attempts: 0 and response body size: -2. It did not fail partway through a response. It never got one started.

PPID 1

The pool for this site is configured for ten children and ten concurrent connections. I expected either zero processes — pool dead, should have respawned — or ten busy ones.

There was exactly one:

  PID   PPID STAT     ELAPSED     TIME %CPU   RSS CMD
793380      1 SN      13:52:58 00:07:31  0.9  7036 lsphp

Read that line carefully, because it contains the entire bug.

PPID 1. The parent is gone. This process was adopted by init. A pool child whose master has died is an orphan, and an orphan is supervised by nothing. Nothing will restart it, nothing will kill it, nothing will notice it.

ELAPSED 13:52:58 against TIME 00:07:31. Fourteen hours of wall clock, seven and a half minutes of CPU. It was not spinning. No load-average alert was ever going to catch this.

STAT SN — interruptible sleep, low priority. The kernel confirms it:

$ grep State /proc/793380/status
State:  S (sleeping)

$ cat /proc/793380/wchan
hrtimer_nanosleep

hrtimer_nanosleep means the process is inside a timed sleep. Some PHP path called sleep() and the value it was handed put the process to bed for what may as well be forever. An API retry backoff, a lock-wait loop, a scraper delay — the specific call matters less than the shape: a wait nobody bounded.

Its open file descriptors closed the case:

$ ls -l /proc/793380/fd
0 -> socket:[598208598]
1 -> /usr/local/.../stderr.log (deleted)
2 -> /usr/local/.../stderr.log (deleted)
3 -> socket:[598208598]

Two descriptors on the listening socket. And stdout and stderr pointing at a log file marked (deleted) — something removed that file while the process held it open, so every byte it wrote for fourteen hours went to an inode with no name. If it printed a fatal error, a stack trace, a reason, that is gone. Never let cleanup delete a log a live process still holds open; rotate it and signal the writer instead.

Why nothing restarted it

Here is the part that generalizes past any one server.

A process supervisor decides whether to start a pool by asking two questions: does the socket file exist, and is the PID in the pid file alive?

srwxr-xr-x 1 ... /tmp/.../siteuser.sock
-rw-r--r-- 1 ... /tmp/.../siteuser.sock.pid

Both answers were yes. The socket file was on disk with the right permissions. The pid file named a process that was alive — sleeping, low priority, adopted by init, but very much running.

So the supervisor concluded the pool was up and never spawned a replacement. Meanwhile the orphan, asleep inside a timer, accepted nothing. Every incoming connection landed on a socket with no reader, waited out the connect timeout, and became a 503.

The pool was neither alive nor dead. It was occupying the identity of a working pool — the socket path and the pid — without doing any of the work. A liveness check that asks "is the PID alive" cannot see this, and neither can anything watching for a crash, because nothing crashed.

Fourteen hours

I went back to the access log and counted requests per hour:

That is the moment of death, and it matches the process start time to within a few minutes. From 20:00 onward the only requests left are bots, retries, and the handful of people who tried, got nothing, and left.

Nineteen requests an hour is not zero, and that is the trap. A "traffic dropped to zero" alert never fires. A daily-total alert compares a bad day against a normal one and shrugs at a 90% drop if the threshold is 95%. The site served errors to every visitor for fourteen hours while the graph showed a line that was low but not flat.

The fix, and it is boring

kill -TERM 793380
rm -f /tmp/.../siteuser.sock /tmp/.../siteuser.sock.pid

Kill the orphan, remove the stale socket and pid file so the supervisor stops believing a pool exists, then send one request. The next request rebuilt the pool by itself. Local response: 200 in 0.171 seconds. From outside: 200 in 0.29 seconds, full page, no error signatures in the HTML. Four healthy children within seconds.

Under ten seconds of repair after fourteen hours of downtime. That ratio is the actual story. The failure was trivial to fix and completely invisible.

What to check instead

Monitor the URL users actually load, and follow redirects to the end. A 302 from a server-level rewrite is not evidence that your application runs. If your monitor reports on the first response in the chain, it is monitoring your rewrite rules.

Assert on the body, not the status code. Match a string that only appears when the application really rendered — a title, a footer, a token in the markup. Byte length works too: a 503 page is a few hundred bytes, a real page is tens of kilobytes.

Alert on relative traffic collapse, not on zero. A drop from 306 to 19 requests in one hour is a far better signal than any absolute floor, and it fires within the hour instead of the next morning.

Treat long-lived pool children as suspect. Pool workers are meant to be recycled. Any child running for hours with almost no CPU deserves a look. ps -eo pid,ppid,etime,time,cmd plus /proc/<pid>/wchan costs nothing in a cron job, and a worker with PPID 1 is an orphan by definition.

Cap every sleep and every outbound call. The root cause under all of this is an unbounded wait. Timeouts on HTTP clients, bounded retry backoffs, lock waits that eventually give up. A worker allowed to sleep forever will eventually do exactly that.

Never delete a log file a process still holds open. Rotate and signal. Otherwise the one artifact that would have told you why is writing into an unlinked inode.

The short version

DNS, TCP, TLS and the certificate can all be perfect while the site returns 503 to every visitor.
External processor is not available means the web server could not get a working connection to the PHP pool. It is a backend problem, not a network or certificate problem.
A pool child whose parent has died gets PPID 1 and is supervised by nothing.
If that orphan keeps the socket and pid file, the supervisor sees a live pool and refuses to spawn a real one. Neither alive nor dead, just occupying the identity.
/proc/<pid>/wchan naming a sleep function, with hours of elapsed time and seconds of CPU, is the confirmation.
Health checks that follow redirects and assert on body content would have caught this in the first minute.

The site was down for fourteen hours. The fix was one signal and two deleted files. Everything expensive about this outage happened in the gap between a process existing and a process working.

Written from production work at Alesta WEB.

Lowercasing "İ" Adds a Character: How Turkish Text Broke a Keyword Router

Mahmut Gündüzalp — Fri, 04 Sep 2026 06:41:21 +0000

A classifier was sending questions to the wrong bucket. Not occasionally. I measured it against twenty real inputs and seven of them landed somewhere they did not belong. Thirty-five percent.

The routing code was thirty lines long and did the obvious thing: lowercase the incoming text, then check whether any keyword for a category appears in it. First match wins.

Every one of those thirty lines was wrong in a way that only shows up in Turkish, and two of the three problems will bite you in any language whose case mappings are not one-to-one.

The symptom

Users type free-form questions. The router picks a topic so that the right prompt and the right reference data get loaded. When it picks wrong, the answer is confidently about the wrong subject, which is worse than no answer.

The reported complaint was "the answers feel shallow." The instinct is to blame the model. The model was fine. It was being handed the wrong context.

Trap 1: lowercasing a dotted capital I produces two code points

Here is the whole thing, measured rather than remembered:

$s = "İş";                              // "work", capital İ
$l = mb_strtolower($s, "UTF-8");

// original      : U+0130 U+015F
// mb_strtolower : U+0069 U+0307 U+015F
// mb_strlen     : 2 -> 3

U+0130 LATIN CAPITAL LETTER I WITH DOT ABOVE has no single-code-point lowercase form in the default Unicode case mapping. It maps to U+0069 LATIN SMALL LETTER I followed by U+0307 COMBINING DOT ABOVE. The string got longer. It still renders as "i̇ş" and looks entirely normal in your terminal, your editor, and your database client.

So this happens:

mb_strpos(mb_strtolower("İşten çıkarılır mıyım", "UTF-8"), "iş");   // false
mb_strtolower("İSTANBUL", "UTF-8") === "istanbul";                  // false

Your keyword is two code points. The haystack has three. No match, no error, no warning. The comparison returns false and the request goes to whatever category happens to be checked next.

The reason this is not an edge case: people capitalize the first word of a sentence. In Turkish a large number of everyday words begin with İ — the words for work, request, relationship, reputation, tender, transaction, need, cancellation. Any of them starting a sentence produces the three-code-point form. The broken form is the common form, and the version your test uses — all lowercase, typed by you — is the rare one.

That is why this survives a green test suite. You wrote "işten çıkarılır mıyım" in the fixture, in lowercase, by hand. It matches. The user wrote "İşten çıkarılır mıyım". It does not.

Unicode normalization will not save you

The reflex is to reach for Normalizer::normalize() and move on. I checked:

lower    : U+0069 U+0307 U+015F
NFC      : U+0069 U+0307 U+015F
NFKC     : U+0069 U+0307 U+015F
casefold : U+0069 U+0307 U+015F

NFC composes a base character with a combining mark when a precomposed code point exists. There is no precomposed "latin small letter i with dot above" — a lowercase i already has a dot, so Unicode never needed one. The sequence stays two code points under every composing form, and casefold() does not touch it either.

NFD is more interesting, because it goes the other way and decomposes everything:

NFD("i̇ş") : U+0069 U+0307 U+0073 U+0327

Now the cedilla under ş is its own code point too. That gives you one rule that handles both problems at once: decompose, then drop every nonspacing mark.

import unicodedata as u

def norm(s: str) -> str:
    s = u.normalize('NFD', s.lower())
    s = ''.join(c for c in s if u.category(c) != 'Mn')
    return s.replace('ı', 'i')      # dotless ı has no combining mark to strip

Measured output:

'İş'         -> 'is'
'İSTANBUL'   -> 'istanbul'
'Değişiklik' -> 'degisiklik'
'İlişki'     -> 'iliski'
'Ölüm'       -> 'olum'

In PHP, where the intl extension is frequently not loaded, the same thing without a normalizer:

function tr_norm(string $s): string {
    $s = mb_strtolower($s, 'UTF-8');
    $s = str_replace("\u{0307}", '', $s);        // the leftover dot from İ
    return strtr($s, [
        'ı'=>'i','ş'=>'s','ğ'=>'g','ü'=>'u','ö'=>'o','ç'=>'c',
        'â'=>'a','î'=>'i','û'=>'u',
    ]);
}

The single most important line is the U+0307 removal. Delete it and the bug is back.

Run your keyword list through the same function, too. Both sides of a comparison must see the same normalization, always. A keyword table normalized at author time and input normalized at request time will drift the moment somebody edits the table by pasting from a document.

The other direction, for readers who do not write Turkish

Turkish has two distinct letters where English has one: dotted İ/i and dotless I/ı. They are different letters, not styling. So the correct Turkish lowercase of I is ı, not i.

This is the well-known "Turkish I problem," and it usually appears as the mirror image of what I described. In Java, .NET, or any locale-sensitive toLowerCase() running under a tr-TR locale:

"INFO".toLowerCase()          // "ınfo" under tr-TR, "info" everywhere else
"FILE".toLowerCase()          // "fıle"

Code that lowercases a protocol keyword, a header name, a file extension, or an SQL identifier and then compares it against an ASCII literal breaks on a machine whose locale happens to be Turkish. The fix there is the opposite of the one above: pass an explicit invariant locale, toLowerCase(Locale.ROOT), and never let ambient locale decide the semantics of a comparison.

So there are two rules, pointing in opposite directions depending on what you are doing:

Comparing protocol tokens and identifiers: force a locale-invariant mapping. Never fold, never guess.
Comparing human text for search or classification: normalize aggressively, on both sides, with a function you control.

Confusing the two is how you end up with a system that is broken in one country and subtly wrong everywhere else.

Trap 2: unbounded substring matching

With normalization fixed, the router still misrouted. The second bug had nothing to do with Unicode: the check was strpos($text, $keyword) !== false, with no word boundary.

Short keywords swallow longer words that merely contain them. Real collisions from the keyword table:

Keyword	Means	Also matches	Which means
`olum`	death	olumlu, olumsuz	positive, negative
`mal`	goods, property	normal, mal**zeme	normal, material
`is`	work	kisi, degisiklik, gise	person, change, counter
`ask`	love	maske	mask
`din`	religion	aydin, din**len	bright, rest

olum was the expensive one. "Olumlu" — "positive" — is among the most common words in any yes/no question. Every such question was routed to the death-and-inheritance category, because that category happened to sit near the top of the map.

Word boundaries in an agglutinative language are not just \b around a fixed string. Turkish stacks suffixes, and stems mutate: "lost" appears as kayıp, kaybettiğim, kaybolan — the stem itself shifts from kayıp to kayb. So match a stem plus a bounded run of following letters:

$pattern = '/(?<!\p{L})' . preg_quote($stem, '/') . '\p{L}{0,6}(?!\p{L})/u';

(?<!\p{L}) and (?!\p{L}) are Unicode-aware boundary assertions — plain \b reasons about \w, whose meaning depends on flags and locale. The {0,6} allows suffixes without letting the stem float in the middle of an unrelated word. And keep a stem list per concept rather than one string: kayıp|kayb|kaybol|kaybet, because sound changes will not occur to you at the keyboard, only in production.

Trap 3: map order is a priority ranking you did not know you wrote

foreach ($map as $topic => $words) {
    foreach ($words as $w) {
        if (matches($text, $w)) return $topic;   // first hit wins
    }
}

Whatever order the array literal happens to be in is your precedence rule. "Will my father's health improve" contains both a family keyword and a health keyword. Family was declared fourth, health sixth, so it went to the family bucket every time.

Two changes fix the class of problem rather than the instance:

Order specific before general, deliberately, with a comment saying so.
Stop returning on first hit. Collect every match, score them, take the best. A reasonable score is the number of distinct stems matched, weighted by stem specificity — a rare six-letter stem is worth more than a three-letter one that appears everywhere.

Scoring also gives you something first-match never can: a confidence number. When the top two topics score within noise of each other, you can fall back to a neutral path instead of committing to a coin flip.

Normalize for matching, never for identity

One caution, because the recipe above is lossy on purpose:

'ILIK'  -> 'ilik'
'ılık'  -> 'ilik'

Two different words collapse to the same key. That is exactly what you want for search and classification, and exactly what you must never do for identity.

If you normalize usernames or email local parts this way before a uniqueness check or a login comparison, you have created an account-collision path: two visually and semantically distinct identifiers now compare equal. The same applies to case-insensitive uniqueness in general — the database collation, the application-level check and the login comparison have to agree on one rule, and that rule should be locale-invariant, not a fuzzy fold.

Fold for relevance. Compare exactly for identity. Keep the two functions in separate files so nobody reuses one for the other by accident.

How to verify, because you cannot eyeball this

Every symptom in this article is invisible on screen. The three-code-point string looks identical to the two-code-point one. You cannot review your way to correctness here; you have to measure.

What I did, and would do again:

Write a table of 20–30 real inputs with the expected category, taken from actual traffic rather than invented. Run it, print the wrong-rate as a number. "Seven of twenty" is a fact you can act on; "seems better now" is not.
Sweep for collisions mechanically. For every keyword shorter than about five characters, generate common words containing it and feed them through the classifier. That is how olum inside olumlu surfaced — nobody was going to catch it by reading.
Print code points, not strings, whenever a comparison fails for no visible reason. printf("U+%04X ") over the characters takes ten seconds and ends the argument.
Re-run the same table after the fix and report the new number. A fix that is not re-measured is a hypothesis.

The short version

mb_strtolower('İ') returns i + U+0307. The string gets longer, looks the same, and stops matching.
NFC and casefold do not collapse it. NFD plus stripping nonspacing marks does.
Normalize the input and the keyword list with the same function.
Match stems with Unicode-aware boundaries, not bare substrings.
Map order is silent precedence. Score instead of returning on first hit.
Fold for search; compare exactly for identity.
Measure the wrong-rate before and after, on real inputs.

None of this is exotic Unicode trivia. It is one line of case mapping and one missing word boundary, and together they were quietly discarding a third of the routing decisions in a system that looked, from the outside, like a model quality problem.

Written from production work at Alesta WEB.

The Word "null" Is Not a Null: How JSON Nulls Turn Into Text on Android

Mahmut Gündüzalp — Sun, 30 Aug 2026 10:39:10 +0000

A user opened our mobile app and saw this on their profile screen:

Membership: null
Points: null
Last payment: null

The same account on the website showed empty fields and a zero. No crash, no error dialog, no red line in the log. The app had received the data, parsed it successfully, and printed the four characters n, u, l, l into a TextView.

The parsing code looked like it had already handled this:

val plan = obj.optString("membership", "")

That is the whole bug. On Android, when membership is JSON null, optString does not return "". It returns the string "null".

Why it does that

org.json on Android represents JSON null with a sentinel object, JSONObject.NULL. It is not Java null — it is a real object that exists specifically so the library can tell "the key is absent" apart from "the key is present and its value is null."

Android's implementation of optString is, in essence:

public String optString(String name, String fallback) {
    Object object = opt(name);
    String result = JSON.toString(object);   // ~ object.toString()
    return result != null ? result : fallback;
}

opt("membership") returns the sentinel. The sentinel is not Java null, so the fallback branch is never reached. toString() is called on it, and JSONObject.NULL.toString() returns — by design, because it has to serialize back to valid JSON — the text "null".

The fallback works exactly as documented for a missing key. It does nothing for a null key. Those are two different states and the API gives them one parameter.

getString has the same outcome. It does not throw on a null value; it hands you "null" as well.

Now compare the neighbours:

obj.optInt("points", 0)        // JSON null -> 0     ✅
obj.optBoolean("active", false) // JSON null -> false ✅
obj.optString("plan", "")       // JSON null -> "null" ❌

optInt and optBoolean route through converters that return Java null for the sentinel, so the fallback fires. Only the string path stringifies the sentinel. This inconsistency is why the bug survives review: three lines that look identical, and one of them behaves differently.

It also depends on which `org.json` you are compiled against

This is worth checking rather than assuming. The reference implementation from json.org — the org.json:json jar you would pull into a plain JVM or server-side project — has a version of the same method that reads roughly:

public String optString(String key, String defaultValue) {
    Object object = this.opt(key);
    return NULL.equals(object) ? defaultValue : object.toString();
}

It checks the sentinel explicitly and returns your default. Same package, same class, same method signature, different answer.

Android ships its own Harmony-derived implementation in the framework, and it wins on the classpath. So a unit test that runs on the JVM with the json.org jar can pass while the device fails. If your test source set is green and your androidTest source set is red on the same assertion, this is a candidate.

Confirm it in ten seconds with an instrumented test rather than trusting either the docs or this article:

@Test fun optStringOnJsonNull() {
    val o = JSONObject("""{"a": null}""")
    assertEquals("", o.optString("a", ""))   // fails on device
}

Where the null came from in the first place

Fixing the client is only half of it, because the response should not have contained nulls for those fields at all.

The API was PHP, and the payload was built from a query that looked something like:

SELECT
    m.plan_name,
    SUM(p.amount)  AS total_paid,
    MAX(p.paid_at) AS last_payment
FROM members m
LEFT JOIN payments p ON p.member_id = m.id
WHERE m.id = ?
GROUP BY m.id

Two separate null factories in one query:

LEFT JOIN with no matching rows. Every column from payments comes back NULL.
Aggregates over an empty set. SUM() and MAX() return NULL, not 0 and not '', when there are no rows to aggregate. COUNT() is the exception — it returns 0 — which is exactly why people assume the others do too.

The website never showed this because the view layer cast on the way out:

echo (int)($row['total_paid'] ?? 0);
echo htmlspecialchars($row['plan_name'] ?? '');

The ?? 0 there is doing real work — ?? catches SQL NULL once PDO has turned it into PHP null. Web output was clean. The JSON endpoint serialized the same row directly, json_encode mapped PHP null to JSON null, and the app printed the word.

This is the general shape: a null that is invisible in one rendering path becomes visible in another. Two consumers of the same data, one of which happens to have a defensive cast in its template.

Why the test accounts never caught it

The seeded test users all had payment history. Every demo account had rows. The only way to produce this response was to log in as a member with nothing — a brand new registration, or an account whose records had been archived.

A user with zero rows is a boundary condition, and it is the one that seed scripts systematically do not create, because seed scripts exist to make screens look populated. If you write fixtures, make one account deliberately empty and put it in the smoke test.

Fix both layers, not one

The tempting move is to fix the server, since that is where the null originates, and declare the client fine. That is wrong for mobile specifically, and here is why.

Old app versions live forever. You control your server. You do not control which build is on a user's phone. A client that renders "null" when handed a JSON null is a permanently loaded gun pointed at every future endpoint, including ones written by someone who has never heard of this bug.

Caches outlive deploys. An HTTP cache, a CDN, or a local response cache can hand the app a pre-fix payload well after the server is patched.

New fields arrive constantly. Server-side normalization is a policy you have to remember to apply. Client-side hardening is a helper you cannot forget to use, because it is the only way to read a string.

So: both.

Server side — normalize once, at the boundary

Recursively convert nulls at the point where the response is serialized, not in each handler:

function json_clean(mixed $value): mixed
{
    if (is_array($value)) {
        foreach ($value as $k => $v) {
            $value[$k] = json_clean($v);
        }
        return $value;
    }
    return $value ?? '';
}

// single exit point for every API response
function api_respond(array $payload, int $status = 200): void
{
    http_response_code($status);
    header('Content-Type: application/json; charset=utf-8');
    echo json_encode(json_clean($payload), JSON_UNESCAPED_UNICODE);
}

One caveat worth stating plainly: this flattens the difference between "no value" and "empty value". For a display API that is what you want. For an API where a client needs to distinguish "field not provided" from "field explicitly cleared" — a PATCH-style contract, for instance — do not do this globally. Fix those fields at the query instead:

COALESCE(SUM(p.amount), 0)   AS total_paid,
COALESCE(m.plan_name, '')    AS plan_name

COALESCE is the honest fix; the recursive normalizer is the safety net under it.

Client side — never call `optString` directly again

Wrap it once, ban the raw call in review:

object Json {

    fun text(obj: JSONObject?, key: String, fallback: String = ""): String {
        if (obj == null || !obj.has(key) || obj.isNull(key)) return fallback
        val value = obj.optString(key, fallback).trim()
        // a server can also send the *string* "null" — from string concatenation,
        // a logging wrapper, or an older client that echoed this bug back
        if (value.isEmpty() || value == "null" || value == "undefined") return fallback
        return value
    }

    fun number(obj: JSONObject?, key: String, fallback: Long = 0): Long {
        if (obj == null || obj.isNull(key)) return fallback
        return obj.optLong(key, fallback)
    }

    fun textOrNull(obj: JSONObject?, key: String): String? =
        text(obj, key, "").ifEmpty { null }
}

isNull(key) is the load-bearing line. It returns true for both Java null and the JSONObject.NULL sentinel, which is the check optString fails to make on your behalf.

Filtering the literal "null" string looks paranoid until you have seen a server do "Welcome, " + name in a template and store the result. Once a "null" has been written to a database by an upstream system, no amount of correctness downstream removes it — you have to strip it at read time.

Then make the raw call unusable:

// build.gradle.kts — fail the build, not the user
// (or use Detekt's ForbiddenMethodCall rule)

A lint rule beats a code review comment, because the review comment only catches the calls written while the reviewer remembers the bug.

What about Gson, Moshi, kotlinx.serialization?

Mostly they handle this correctly, with one trap of their own.

Moshi and kotlinx.serialization are aware of Kotlin nullability. Declare val plan: String as non-null and feed it JSON null, and they throw at parse time — a loud, immediate, correctly-located failure. That is the behaviour you want.

Gson does not. It constructs objects through unsafe reflection and bypasses Kotlin's null checks entirely, so a JSON null lands in a field the type system swears is non-null. Nothing fails at parse time. The NPE happens later, somewhere unrelated, and the stack trace points at the innocent code that touched the field rather than the parser that lied about it. That is arguably worse than "null" on screen, because at least "null" on screen tells you where it came from.

If you are on Gson, declare the fields nullable (String?) and handle the nulls explicitly. Fighting for non-null fields you cannot enforce buys nothing.

org.json remains everywhere despite all three of these, for a simple reason: it is in the Android framework, weighs nothing, and needs no annotation processor. Small apps and old modules keep using it. It works fine — as long as you know that its string path does not have a null story.

The five-minute audit

Worth running on any app that talks to an API you also control:

grep -rn "optString(" app/src — every hit is a candidate. Replace with a helper that calls isNull first.
grep -rn "getString(" app/src on JSONObject receivers — same problem, no fallback parameter to give false comfort.
In the API, find aggregates without COALESCE: SUM(, MAX(, MIN(, AVG( — and every LEFT JOIN selecting columns from the right-hand table.
Create one test account with zero related rows and open every screen with it. This finds more null bugs in ten minutes than reading code for a day.
grep your database for the literal string 'null' in text columns. If you find any, something upstream has already been concatenating nulls into text, and that data needs cleaning independently of any code fix.

The underlying lesson is not about one library method. It is that null crosses system boundaries badly. SQL NULL, PHP null, JSON null, Java null, Kotlin null?, and a sentinel object that stringifies to "null" are six different things wearing the same name, and every hop between two of them is a place where one quietly becomes another. Put the conversion in one function per boundary, and you only have to be right once.

I build and run news, e-commerce and membership platforms at alestaweb.com. This one took an afternoon and a screenshot from a user who assumed "null" was our membership tier.

MySQL Will Quietly Eat Your Data If You Let It: Truncation, Charsets, and the 1366 You Never See

Mahmut Gündüzalp — Sun, 23 Aug 2026 08:30:21 +0000

A WhatsApp button on a shop was dialling the wrong number. Not a formatting problem — a genuinely different number, three digits short of the one in the admin panel.

The admin panel had saved it. The page was reading it. Nothing errored. The column was varchar(10), the number arrived with a country code, and MySQL had done exactly what it was configured to do: chop the value to fit and carry on.

That column had been wrong for months. Nobody noticed because nothing anywhere in the stack raised its voice.

Here is the family of bugs that behaves this way, how to find them in a database you inherited, and why the obvious fix — "just turn on strict mode" — creates a different bug if you do only that.

Part 1: The truncation

Run this on any MySQL or MariaDB instance:

SELECT @@GLOBAL.sql_mode, @@SESSION.sql_mode;

If neither contains STRICT_TRANS_TABLES or STRICT_ALL_TABLES, then for every INSERT and UPDATE your server is in the business of making data fit, not of telling you it doesn't.

What that means concretely:

CREATE TABLE t (phone VARCHAR(10));
INSERT INTO t VALUES ('905551234567');
SHOW WARNINGS;
-- Level: Warning  Code: 1265  Message: Data truncated for column 'phone' at row 1

SELECT phone FROM t;
-- 9055512345

Note the level. Warning, not error. The statement succeeded. affected_rows is 1. PDO in exception mode throws nothing, because there is nothing to throw — MySQL considers this a completed statement. Your ORM reports success. Your integration test asserting "the row exists" passes. The value is wrong.

The same mechanism applies well beyond strings:

You wrote	Column	Non-strict result
`'905551234567'`	`VARCHAR(10)`	`'9055512345'`
`300`	`TINYINT`	`127`
`'2026-02-31'`	`DATE`	`'0000-00-00'`
`''`	`INT NOT NULL`	`0`
`12.999`	`DECIMAL(4,2)`	`13.00`

Every one of those is a silent, permanent difference between what your application believed and what your database holds. And unlike a crash, there is no timestamp to correlate against — you cannot tell from the row when it happened or how many rows before it went the same way.

Finding the damage in a database you inherited

You cannot recover truncated values — the tail is gone. But you can find the columns where it is happening, which is what matters going forward. Truncated values pile up at exactly the column limit, so the suspects are the columns where real rows sit at the maximum length:

-- 1. list the string columns and their limits
SELECT TABLE_NAME, COLUMN_NAME, CHARACTER_MAXIMUM_LENGTH AS max_len
FROM information_schema.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
  AND DATA_TYPE IN ('varchar', 'char')
  AND CHARACTER_MAXIMUM_LENGTH <= 64      -- short columns are where this bites
ORDER BY CHARACTER_MAXIMUM_LENGTH;

Then, per interesting column:

SELECT COUNT(*) AS at_the_limit
FROM settings
WHERE CHAR_LENGTH(phone) = 10;

A handful of rows sitting exactly at the limit can be a coincidence. A majority of rows sitting exactly at the limit is not a coincidence, it is a report. In our case the count was every row that had ever been saved with a country code.

Two details that matter when you run this:

Use CHAR_LENGTH(), not LENGTH(). LENGTH() counts bytes; CHAR_LENGTH() counts characters. On multi-byte text they disagree, and VARCHAR(n) limits characters, not bytes.
Do this on a replica or a restored dump if the table is large. CHAR_LENGTH() in a WHERE clause means a full scan.

Part 2: Strict mode is necessary and not sufficient

The obvious response is to turn strict mode on. Do it — but understand what you are buying.

Strict mode does not repair the mismatch between what your application sends and what the column accepts. It changes how you find out about it: instead of quietly storing a wrong value, MySQL raises error 1406 (Data too long for column) and your code, which never expected a write to fail, hands the user a 500.

I have seen exactly this on an admin form. Strict mode was correctly enabled; the title field in the form had no maxlength and no server-side length check; the column was varchar(150). Someone pasted a long headline. Instead of a validation message the editor got a blank error page, and — worse — a story they thought they had saved and hadn't.

That is still better than silent corruption, because it is loud and it is immediate. But the actual fix is at the boundary:

$rules = [
    'title' => ['max' => 150],   // same number as the column, in one place
    'phone' => ['max' => 20],
];

foreach ($rules as $field => $rule) {
    if (mb_strlen($input[$field] ?? '') > $rule['max']) {
        return back()->withError("{$field}: at most {$rule['max']} characters.");
    }
}

Three habits that keep this honest:

Validate before you write, with the column's real limit. Generate the limits from information_schema if you can — a hand-copied number drifts the first time someone runs an ALTER.
Turn strict mode on so the boundary check has a backstop. Validation you wrote can be bypassed; the database is the last honest party in the chain.
Widen the column when the data is legitimately bigger. VARCHAR(10) for a phone number was never right. Truncation is a symptom; a wrong schema is the disease.

Part 3: Charsets, where "it worked on the old server" lives

A database was copied to a new server. Everything imported. Row counts matched. A week later somebody noticed that Turkish characters in older records had turned into question marks in some tables and not others.

The mistake was trusting the database default:

SHOW CREATE DATABASE app;
-- CHARACTER SET utf8mb4  ← looks fine, means almost nothing

The database default applies to newly created tables that do not specify their own. It says nothing about the columns you already have. The columns are where the truth is:

SELECT TABLE_NAME, COLUMN_NAME, CHARACTER_SET_NAME, COLLATION_NAME
FROM information_schema.COLUMNS
WHERE TABLE_SCHEMA = 'app'
  AND CHARACTER_SET_NAME IS NOT NULL
  AND CHARACTER_SET_NAME <> 'utf8mb4';

If that returns rows, you have a mixed-charset database, and every dump/restore across it is a chance to lose bytes.

Two more charset facts worth having in your head:

utf8 in MySQL is not UTF-8. The alias utf8 (historically utf8mb3) stores at most three bytes per character, so it cannot hold anything outside the Basic Multilingual Plane — emoji, some CJK extensions, some historic scripts. Insert one into a utf8 column and, in non-strict mode, MySQL truncates the value at the emoji and keeps whatever came before. utf8mb4 is real UTF-8. Use it everywhere, including the connection charset.

The connection charset is part of the pipeline. Your column can be utf8mb4, your data can be perfect, and your terminal or client can still show you mojibake because the session negotiated something else. Which leads to the most useful debugging habit in this whole article:

When you think data is corrupted, check the bytes

SELECT title, HEX(title), LENGTH(title), CHAR_LENGTH(title)
FROM articles WHERE id = 42;

HEX() is the ground truth. It is not affected by your client, your terminal font, or the connection charset. I have twice now "found" data loss that turned out to be a display artefact of the command-line client, and both times HEX() settled it in one query: the bytes were intact, the rendering was not.

The tell for genuine UTF-8 content is LENGTH() > CHAR_LENGTH(). If they are equal on text you know contains non-ASCII characters, something upstream already flattened it.

Part 4: The application-side version of the same bug

Databases are not the only layer that cuts strings. This is a summary excerpt in PHP:

$summary = substr($body, 0, 200);   // wrong

substr() counts bytes. On UTF-8 text, byte 200 lands in the middle of a multi-byte character about half the time, and you send MySQL a string ending in half a character. Then either:

the column is utf8mb4 and strict mode is on → error 1366, Incorrect string value, and you get a 500 from what looked like a formatting line; or
strict mode is off → MySQL drops the invalid tail silently, and your summaries are quietly one character shorter than you think, sometimes ending in a replacement glyph.

The fix is one letter and three characters:

$summary = mb_substr($body, 0, 200);   // counts characters

Whenever you see strlen, substr, strtoupper, or str_pad applied to user text, treat it as a bug report waiting to happen. Error 1366 in a log almost always traces back to one of them.

The checklist

Run these on any project you did not set up yourself. They take about ten minutes together.

SELECT @@GLOBAL.sql_mode; — is STRICT_TRANS_TABLES present? If not, plan to enable it after auditing lengths, not before.
List varchar columns with small limits, then count rows sitting exactly at the limit. That count is your truncation report.
List columns whose CHARACTER_SET_NAME is not utf8mb4. Fix them before the next migration, not during it.
Confirm the connection charset your application actually negotiates — the DSN, not the config file you hope it reads.
Grep the codebase for byte-based string functions applied to user content, and replace them with the mb_ versions.
When you suspect corruption, run HEX() before you conclude anything. Half the time the data is fine and the client is lying.

The theme across all of this: the database is willing to accept an approximation of what you gave it, and by default it will not argue. Make it argue. Then make sure your code is ready to hear it.

I build and run news and e-commerce platforms at alestaweb.com. Every example above cost somebody a real afternoon.

Your Uptime Monitor Says 200. Your Contact Form Has Been Dead for Three Days.

Mahmut Gündüzalp — Sun, 23 Aug 2026 08:27:02 +0000

The contact form on a company site I look after stopped producing leads. Not "fewer leads" — zero, for three days. No alert fired. The uptime monitor was green the entire time. The status page had a nice unbroken bar.

The monitor was checking the homepage. The homepage was fine. /contact had been returning a fatal error since a Tuesday afternoon deploy.

This is the most boring class of outage there is, and it is the one that costs the most, because nothing tells you it is happening. Below is what actually broke, the four ways a site can be broken while still answering 200, and the small set of checks I run now instead of pinging the root URL.

What broke

The site loads a list of helper files at boot. During an unrelated edit, that list was rewritten, and one entry did not survive the rewrite — a helper that supplied data to a few templates.

Most pages never touch that helper, so they kept rendering. One page called it in a template. That page fataled. In production, with display errors off, a fatal means an empty response body or a generic error page — depending on how the stack is configured, sometimes with a 500 status, sometimes with the web server's own branded page and a status you did not choose.

The error log had the answer on line one, at the exact timestamp of the deploy. Nobody looked, because nothing said to look.

Three days of a form that a campaign was actively driving traffic to. The traffic arrived. The page did not.

Four ways to be broken and still return 200

If your health check is curl -o /dev/null -w "%{http_code}" https://example.com/ you are testing one route, one time, for one property. Here are the failure modes that sail straight through it.

1. The route you don't check. A site is not a URL, it is a few dozen to a few hundred routes. Homepages are the least likely page to break, because everything touches them and everyone looks at them. The pages that break are the ones with one unusual dependency: the form page, the search page, the report that joins four tables.

2. The page renders, but a part of it silently didn't. Templates are forgiving by design. A partial that throws inside a try/catch, a widget whose data source returned an empty array, a block that renders only when a config key exists — none of these change the status code. The page looks right at a glance and is missing the thing that made it worth serving.

I found one of these in the same audit: a function that emitted structured data (Article, BreadcrumbList) existed, was correct, and was called from nowhere. Thirty news URLs had been building JSON-LD that never reached the HTML. Status: 200. Value delivered: none.

3. The page is fine and the form is not. This one is hard to see. A full-page HTML cache was turned on. HTML caches are dumb by nature: they store bytes. Those bytes included a CSRF token. Every visitor got the first visitor's token, every POST failed validation, and the failure looked to the user like a page reload with no message. Every page on the site: 200. Every form on the site: broken.

4. The error page returns 200. Custom error handlers that render a friendly "something went wrong" view and forget to set the status. Frameworks that catch late and fall back to a template. Reverse proxies serving a static maintenance page with the default status. Your monitor sees 200 and a body of some length, and reports health.

What to assert instead

The fix is not a bigger monitoring product. It is asserting the things you actually care about. Four checks, in increasing order of value.

Check every route, not one route

You already have the route list — it is in your router, your sitemap, or both. Walk it.

#!/usr/bin/env bash
# routes.txt: one path per line, from the sitemap or the router
FAIL=0
while read -r path; do
  code=$(curl -s -o /tmp/body -w '%{http_code}' "https://example.com${path}")
  size=$(wc -c < /tmp/body)
  if [ "$code" != "200" ] || [ "$size" -lt 2000 ]; then
    echo "FAIL ${path} status=${code} bytes=${size}"
    FAIL=1
  fi
done < routes.txt
exit $FAIL

The size floor is doing real work here. A fatal that returns 200 with an empty or near-empty body is caught by bytes, not by status. Pick the floor from your own smallest legitimate page, then subtract a little.

Assert content, not just bytes

One string per route, chosen to be the thing that page exists to do. The form page must contain a <form and a submit button. The article page must contain an <h1. The page with structured data must contain application/ld+json.

grep -q '<form' /tmp/body || echo "FAIL ${path}: no form in body"
grep -qi 'fatal error\|<b>Warning</b>\|stack trace\|Undefined variable' /tmp/body \
  && echo "FAIL ${path}: error signature in HTML"

That second grep catches the inverse case: the page that renders and prints a warning into the output where a visitor — or a crawler — can read it.

Diff the error log around every deploy

This is the cheapest high-value check in the list and almost nobody does it. Record the log size before the deploy, read the tail after.

BEFORE=$(stat -c%s /path/to/error.log)
# ... deploy ...
sleep 30
AFTER=$(stat -c%s /path/to/error.log)
if [ "$AFTER" -gt "$BEFORE" ]; then
  tail -c $((AFTER - BEFORE)) /path/to/error.log
  echo "New errors appeared during deploy — investigate before walking away."
fi

In our incident this would have printed the missing helper, by name, thirty seconds after the deploy that caused it. The information was there the whole time. Nothing surfaced it.

Round-trip the form

The only way to know a form works is to submit it. A synthetic submission is about twenty lines and can run hourly:

<?php
// 1. Fetch the page like a browser would, keeping cookies.
$ch = curl_init('https://example.com/contact');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_COOKIEJAR      => '/tmp/probe-cookies',
    CURLOPT_COOKIEFILE     => '/tmp/probe-cookies',
]);
$html = curl_exec($ch);

// 2. Pull the CSRF token out of the markup, exactly like a browser.
preg_match('/name="_token" value="([^"]+)"/', $html, $m);
$token = $m[1] ?? null;
if (!$token) { exit("FAIL: no CSRF token on the page\n"); }

// 3. Submit with a marker you can find and delete later.
$marker = 'probe-' . date('YmdHi');
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query([
    '_token'  => $token,
    'name'    => $marker,
    'email'   => 'probe@example.com',
    'message' => 'automated probe, safe to delete',
]));
curl_exec($ch);

// 4. Assert the effect, not the response.
$found = (int) $pdo->query(
    "SELECT COUNT(*) FROM leads WHERE name = " . $pdo->quote($marker)
)->fetchColumn();

echo $found === 1 ? "OK\n" : "FAIL: submitted, nothing stored\n";

Step 4 is the point of the whole thing. Not "did the POST return 200" — did a row appear. That single assertion catches the frozen-token case, the silently failing mail transport, the validation rule someone tightened, and the database column that quietly truncated the input.

Give the probe an obvious marker and delete its rows on a schedule. Yes, this writes to production. That is the trade: a handful of tagged test rows a day against not knowing your lead form is dead.

Why this keeps happening

Three reasons, and none of them are laziness.

Monitoring defaults are shaped by hosting, not by the product. A default check answers "is the machine up and serving". That was the right question when a machine going down was the common failure. The common failure now is code that is up and wrong.

Green is a feeling, and feelings are expensive to give up. An unbroken status bar makes people stop looking. Silence should mean "the assertions passed". In practice it means "nothing asserted anything".

Silent failure has no symptom by definition. A crash gets escalated in an hour. A form that swallows submissions gets noticed when someone asks why the pipeline is empty — which, in our case, took three days, and only because someone went looking for a different number.

The short version

A status code is a transport fact. It says the request reached something. It does not say the page did its job.
Check every route, not the homepage. Your sitemap already lists them.
Assert one meaningful string per route, and grep the body for error signatures while you are there.
Diff the error log across every deploy, automatically. Cheapest check here.
Round-trip the forms that make you money, and assert the stored row, not the response.
Anything that renders "sometimes" — structured data, widgets, conditional blocks — needs an output assertion, because code review will not catch a function that is never called.

None of this is sophisticated. The whole set is a cron job and about a hundred lines. The reason to write it is that the alternative is finding out from a person, and by then you have already paid for the traffic.

I build and maintain news and e-commerce platforms at alestaweb.com. Most of what I know about monitoring came from outages exactly this dumb.

The Query That Ate 75% of Our Database CPU: A MySQL Full-Text Post-Mortem

Mahmut Gündüzalp — Thu, 06 Aug 2026 22:11:21 +0000

A news platform I maintain started serving pages in six seconds. Load average sat at 16 on a 12-core box for hours. Nothing had been deployed. Traffic was up, but not 10x up.

The cause turned out to be a single SELECT that looked completely reasonable — the kind of query that passes code review, works fine on a 5,000-row table, and quietly becomes a wrecking ball at 80,000 rows.

Here is the whole investigation: how I found it, why it was slow, what the fix was, and the three unrelated things I learned along the way.

Symptom first

The obvious metrics:

Load average 16.7 on 12 cores
MySQL at 92% CPU, resident memory 9.8 GB out of 15 GB
4.8 GB pushed into swap
Time to first byte on article pages: 6.3 seconds

The tempting move here is to start tuning. Bump the buffer pool, add more cache, blame the bots. I've done that before and it's mostly a way of not finding the bug.

Find the query, not a query

The mistake I see most often at this stage is taking one snapshot of the process list, spotting something slow, and declaring victory. One snapshot tells you what was running at one instant. It doesn't tell you what dominates.

Sample it instead:

for i in $(seq 1 20); do
  mysql -N -e "SELECT info FROM information_schema.processlist
               WHERE command='Execute' AND info IS NOT NULL"
done | sed -E 's/.*MATCH.*/RELATED-ARTICLES/' | sort | uniq -c | sort -rn

Twenty cheap samples, bucketed by shape. The result:

    202 RELATED-ARTICLES
     41 other article queries
     21 rate-limit counter

202 of 268 active queries — 75% — were the same statement. That's not a slow query problem, that's a "one feature is eating the server" problem.

The query

It powers the "related articles" block under every story:

SELECT a.id, a.title, a.summary, a.image, a.published_at,
       MATCH(a.title, a.summary) AGAINST(? IN NATURAL LANGUAGE MODE) AS relevance
FROM articles a
LEFT JOIN categories c ON a.category_id = c.id
WHERE MATCH(a.title, a.summary) AGAINST(? IN NATURAL LANGUAGE MODE)
  AND a.id != ? AND a.status = '1'
ORDER BY (relevance / (DATEDIFF(NOW(), COALESCE(a.sort_date, a.published_at)) / 30 + 1)) DESC
LIMIT 6

Read that ORDER BY again. It divides the relevance score by the article's age in months, so a strong match from 2019 loses to a decent match from last week. It's a genuinely nice ranking idea. Freshness matters in news.

And the search term passed in? The current article's entire title plus the first 200 characters of its summary.

Why it's expensive

Two things compound.

Natural language mode is not a filter, it's a scorer. Feed it a long string and it matches on any meaningful token in that string. I measured it:

SELECT COUNT(*) FROM articles
WHERE MATCH(title, summary) AGAINST('<full title + 200 chars of summary>'
                                    IN NATURAL LANGUAGE MODE);
-- 16,141

16,141 rows matched out of 80,836 — about 20% of the table. A long natural-language term is a very wide net. Every extra common word widens it further.

The ORDER BY cannot use an index. It sorts on an expression computed per row. The optimizer has no choice:

+----------+----------------+----------------------------------------------+
| type     | key            | Extra                                        |
+----------+----------------+----------------------------------------------+
| fulltext | ft_title_summ  | Using where; Using temporary; Using filesort  |
+----------+----------------+----------------------------------------------+

Using temporary; Using filesort on a fulltext scan is the whole story. To return 6 rows, the server:

scores ~16,000 rows,
materialises them into a temporary table,
computes relevance / (age/30 + 1) for every one of them,
sorts all 16,000,
throws away 15,994.

Measured cost: ~0.22 s of CPU per call. On every article page view. With half the traffic coming from crawlers walking the archive, each hitting a different article — so per-article caching had a near-zero hit rate for exactly the traffic causing the load.

At a handful of article views per second, that one query alone wants more cores than the machine has.

Fix 1: stop searching with a paragraph

The search term should be the subject of the article, not the article. I extract meaningful words from the title only — drop stopwords, drop anything under three characters, cap at eight words:

function searchTermFromTitle(string $title): string
{
    $stop = ['and','the','with','for','from','that','this','was','are','has'];
    $clean = preg_replace('/[^\p{L}\p{N}\s]+/u', ' ', $title);
    $clean = trim(preg_replace('/\s+/u', ' ', $clean));

    $picked = [];
    foreach (explode(' ', $clean) as $word) {
        $lower = mb_strtolower($word, 'UTF-8');
        if (mb_strlen($lower, 'UTF-8') < 3 || in_array($lower, $stop, true)) {
            continue;
        }
        $picked[] = $word;
        if (count($picked) >= 8) break;
    }
    return $picked ? implode(' ', $picked) : $clean;
}

Match count dropped from ~16,000–58,000 (it varied wildly by article) to ~2,500–13,000. Better, not solved.

A note for non-English text: build the stopword list for the language you actually store. Mechanically reusing an English list on Turkish, German or Finnish content will either strip nothing or strip the wrong things, and morphology means a naive list misses inflected forms.

Fix 2: rank in two stages

The real problem isn't the match count, it's sorting all matches by an expression. So don't. Take the top N by raw relevance — which the fulltext index can drive with a bounded sort — then apply the freshness weighting to those N in an outer query:

SELECT t.id, t.title, t.summary, t.image, t.published_at, t.relevance
FROM (
    SELECT a.id, a.title, a.summary, a.image, a.published_at,
           MATCH(a.title, a.summary) AGAINST(? IN NATURAL LANGUAGE MODE) AS relevance,
           COALESCE(a.sort_date, a.published_at) AS rank_date
    FROM articles a
    LEFT JOIN categories c ON a.category_id = c.id
    WHERE MATCH(a.title, a.summary) AGAINST(? IN NATURAL LANGUAGE MODE)
      AND a.id != ? AND a.status = '1'
    ORDER BY relevance DESC
    LIMIT 60
) t
ORDER BY (t.relevance / (DATEDIFF(NOW(), t.rank_date) / 30 + 1)) DESC
LIMIT 6

The expensive sort now runs over 60 rows instead of 16,000. The derived table has a LIMIT, so it can't be merged away — it's materialised, which is precisely what I want here.

Ranking quality barely moves. A result that wins after freshness weighting was already a strong relevance match; it was never sitting at position 4,000.

Results

Measured against real rows, old query vs new, five articles:

	Before	After
5 queries, total	2.571 s	0.572 s
Share of active queries	75%	16%
Load average	16.7	7.4
TTFB, worst page	6.34 s	1.91 s

4.5x on the query. And no caching layer involved — this is the same work, arranged so the database isn't asked to sort a haystack to hand back six needles.

Three things I learned that weren't the bug

InnoDB does not give space back. I pruned about 2.9 million rows from some log tables. data_free went up by 270 MB and the files didn't shrink. Deleted rows leave free pages inside the tablespace, reusable by that table but not returned to the OS. OPTIMIZE TABLE rebuilds it. If your cleanup job only deletes, your disk usage will never reflect it.

A dynamic variable can fail silently. innodb_buffer_pool_size is dynamic on modern MariaDB, so I resized it live. No error. The value didn't change. The reason: innodb_buffer_pool_chunk_size read as 0, and resizing works in chunk units. Always read the variable back after you set it — don't trust the absence of an error.

Long-lived MySQL processes bloat. That instance had been up 85 days with a 1 GB buffer pool and was holding 9.8 GB resident, 4.8 GB of it swapped. After a restart with a properly sized pool: 1.96 GB resident, swap essentially empty, and 8 GB handed back to the OS. Per-connection buffers and temp-table churn accumulate; a restart is sometimes the honest fix.

What I'd take away

Sample the process list, don't snapshot it. The dominant query and the slowest query are usually different queries, and the dominant one is what's hurting you.
Using temporary; Using filesort next to a LIMIT 6 is a smell. It means the server built the whole set to return a fraction of it.
Natural language full-text search scales with term length. A search term built from a whole paragraph is a query against a fifth of your table.
Fix the shape before you add cache. Caching this would have hidden it from users while the crawler traffic — cache-missing by construction — kept the CPU pinned.

The bug had been there for a long time. It only became visible when the archive got big enough and the crawlers got busy enough for those two curves to cross.

I build news and e-commerce platforms in PHP at Alesta WEB, an independent software company running since 2005.

Self-Hosted MCP: Building a Model Context Protocol Server in PHP

Mahmut Gündüzalp — Thu, 23 Jul 2026 11:11:26 +0000

Most of the Model Context Protocol tutorials you'll find are written in TypeScript or Python, because those are the languages with official SDKs. That leaves a fair number of us — people maintaining PHP applications that hold years of business data — wondering whether the protocol is even available to us.

It is. MCP is a wire protocol, not a library. If your language can read a line from standard input and write JSON back, you can implement a server in it. This post walks through what the protocol actually asks of you, what a PHP implementation looks like in outline, and — the part I underestimated — what changes when you expose an internal system to a model.

What MCP actually is

The Model Context Protocol is an open standard for connecting AI assistants to external systems: your data, your tools, your APIs. The problem it solves is combinatorial. Before a standard existed, every assistant needed a bespoke integration with every data source. MCP defines one interface so that any compliant client can talk to any compliant server.

Underneath, it's JSON-RPC 2.0. Requests carry a jsonrpc version, a method, a params object, and an id; responses carry the matching id and either a result or an error. Notifications are requests without an id and expect no reply. If you've implemented a JSON-RPC service before, you already know 80% of the transport story.

A server exposes three kinds of primitive:

Tools — actions the model can invoke. Each has a name, a description, and a JSON Schema describing its inputs. This is the primitive that does something: query a database, create a record, send a request. The model chooses when to call them.
Resources — data the model can read, addressed by URI. Files, records, generated documents. These are for context, not action, and the client generally decides what to pull in.
Prompts — reusable prompt templates the user can invoke deliberately, often surfaced in the client's UI as a slash command or menu item.

The distinction between tools and resources matters more than it first appears. A rough rule: tools are model-controlled, resources are application-controlled. If the model should decide whether to fetch something, make it a tool. If the user or the host application decides, make it a resource.

The two transports

MCP defines two standard transports, and picking the right one is the first architectural decision.

stdio. The client launches your server as a subprocess and talks to it over standard input and output — newline-delimited JSON, one message per line. This is the simplest possible setup: no ports, no HTTP server, no authentication layer, because the only thing that can talk to your process is the parent that spawned it. It's the right choice for anything running on the same machine as the client.

Two rules with stdio, both easy to break in PHP:

Never write anything but protocol messages to stdout. A stray echo, a var_dump left in from debugging, or a PHP warning printed to stdout corrupts the message stream and the client will fail to parse it. Send diagnostics to stderr instead, which the client typically forwards to a log.
Turn off output buffering and flush after every write, or your responses sit in a buffer while the client waits.

Streamable HTTP. The server runs as an ordinary HTTP endpoint. The client POSTs JSON-RPC messages to it; the server replies with either a single JSON response or a stream of server-sent events when it needs to push multiple messages for one request. This is the transport for a server that runs somewhere other than the user's machine — which, for most PHP shops, is the interesting case, because that's the deployment model we already know how to operate.

(An older HTTP+SSE transport exists in earlier revisions of the spec. New work should target Streamable HTTP.)

The handshake

Whichever transport you pick, the conversation starts the same way. The client sends initialize with the protocol version it speaks and the capabilities it supports. Your server replies with its own protocol version, its capabilities, and its name and version. The client then sends an initialized notification, and normal operation begins.

Capabilities are how the two sides negotiate. If your server doesn't implement resources, you don't advertise the resources capability, and a well-behaved client won't call resources/list. Don't advertise what you haven't built.

After the handshake, the methods that matter are predictable:

Method	What it does
`tools/list`	Return the tools you expose, with descriptions and input schemas
`tools/call`	Execute one tool with the given arguments, return its result
`resources/list`	Return available resources with their URIs
`resources/read`	Return the contents of one resource
`prompts/list`	Return available prompt templates
`prompts/get`	Return one filled-in prompt

A minimal but genuinely useful server is initialize plus tools/list plus tools/call. Everything else is optional.

What the PHP side looks like

The structural shape, transport-independent:

read a JSON-RPC message
  → dispatch on `method`
  → build a result (or an error)
  → write the response with the same `id`

For stdio, that's a loop over fgets(STDIN), json_decode, a match on the method name, and fwrite(STDOUT, json_encode($response) . "\n"). For Streamable HTTP, it's a single endpoint that decodes the request body and returns the encoded response. The dispatch layer in the middle is identical; only the read and write ends change. Write it that way from the start and you can support both from one codebase.

Three PHP-specific things worth knowing before you start:

Tool schemas. Every tool needs a JSON Schema for its inputs. Hand-writing those as nested arrays gets tedious fast. Deriving them from something you already maintain — a validation ruleset, a DTO, a set of typed constructor parameters read via reflection — keeps the schema and the actual implementation from drifting apart. Schema drift is the single most common cause of "the model keeps calling the tool wrong."

Error handling. Distinguish two kinds of failure. A malformed request or an unknown method is a protocol error: return a JSON-RPC error object. A tool that ran but failed — record not found, validation rejected the input — is a tool error: return a normal result with the error flag set and a human-readable message. The distinction matters because the second kind goes back to the model, which can read the message and adjust. A protocol error just tells it something broke. Convert PHP exceptions into the second kind wherever the failure is something the model could plausibly recover from.

Long-running work. PHP's request-per-process model is a good fit for stdio (the process lives as long as the session) and a slightly awkward one for HTTP if a tool takes minutes. Keep tool calls short. If a tool kicks off something slow, return a job identifier immediately and expose a second tool that reports status. Models handle that pattern well; they handle a request that times out badly.

Connecting it to Claude

Once the server runs, there are three broad ways to reach it.

Local, over stdio. Desktop and CLI clients — Claude Desktop, Claude Code, and various IDE integrations — let you register a server by specifying a command to run and its arguments (php, plus the path to your server script), and they manage the subprocess for you. This is the fastest way to get from "it responds to tools/list" to "I'm using it."

Remote, over HTTP. A Streamable HTTP server deployed at a URL can be registered with clients that support remote connections. The Claude API also has an MCP connector: you declare the server's URL in the request, and the API makes the connection server-side, so the model can call your tools without you writing a client loop. Note that hosted MCP servers generally authenticate with OAuth bearer tokens rather than a service's own native API key — those are different auth systems, and assuming the latter works is a common early stumble.

From your own code. If you're building an agent rather than using an existing client, most AI SDKs can convert MCP tool definitions into their native tool format, so an MCP server becomes a source of tools for a loop you control.

Whichever route, debug over stdio first. The failure modes are simpler: no TLS, no auth, no proxy, no CORS. Get the protocol right, then move it to HTTP.

The part I underestimated: exposure

Here's the thing that changes when you put an MCP server in front of an existing system. Every tool you expose is a capability granted to a model that is, at least in part, steered by text it reads from the outside world. If the model can be persuaded to call your tool, your tool runs.

The practical consequences:

Scope tools narrowly. A generic run_query tool that accepts arbitrary SQL is the most convenient thing to build and the worst thing to ship. Prefer specific tools with typed parameters — find_customer_by_email, list_orders_in_range — and validate every argument server-side as if it came from an anonymous HTTP request. It did, effectively.

Read and write are different risk classes. Read-only tools have a disclosure risk. Write tools have a this actually happened risk. Separate them, and put anything destructive or irreversible behind an explicit confirmation in the host application rather than trusting the model to be careful.

The description is part of the security surface. Tool descriptions are instructions the model reads. A vague description invites the model to call a tool in situations you didn't intend. Being prescriptive about when a tool should be used — not just what it does — measurably improves both correctness and safety.

Don't leak what you didn't mean to expose. Return the fields the task needs, not whole records. An internal note, a cost price, or a personal phone number that shouldn't be in a customer-facing answer shouldn't be in a tool result either. Filter at the server, not in the prompt.

Log every call. Tool name, arguments, caller, outcome. When someone asks "why did it do that," the log is the only answer you'll have.

None of this is exotic — it's the same discipline you'd apply to a public API endpoint. The difference is that the caller is a language model rather than a developer reading your docs, so ambiguity gets resolved in ways you didn't anticipate rather than surfacing as a support ticket.

Worth it?

For a PHP application sitting on years of accumulated business data, MCP is the cheapest bridge I've found between that data and an assistant that can actually reason about it. There's no SDK to wait for. It's JSON-RPC over a pipe or an HTTP endpoint — both things PHP has done well for twenty years.

Start with three read-only tools that answer questions someone in your organization asks weekly. Ship it over stdio to one person. See what they actually ask for. That's a far better spec than anything you'd design up front.

If you've built an MCP server in a language without an official SDK, I'd be curious what tripped you up — the transport, the schemas, or the scoping.

From Generic CMS to Purpose-Built: What 200 Site Migrations Taught Me

Mahmut Gündüzalp — Thu, 23 Jul 2026 11:09:31 +0000

Migrating a live website to a new content management system is one of those jobs that looks like a data transfer and turns out to be an exercise in risk management. The database part is usually the easy half. The hard half is that a working site has years of accumulated meaning attached to it — URLs search engines memorised, editors' habits, a content model that drifted far from whatever it was on day one.

Over the past years I've moved a few hundred sites — mostly news portals and online stores — off general-purpose, plugin-assembled CMS setups onto purpose-built platforms. Some of those migrations were clean. A few taught me things the hard way. This is the accumulated checklist, aimed at anyone about to do their first serious one.

Why sites migrate at all

Nobody migrates a CMS for fun. The trigger is almost always one of four things, and it's worth naming yours honestly before you start, because it determines what "success" means.

Plugin sprawl. A site starts generic and grows domain-specific through add-ons: one plugin for the ticker, one for the photo gallery, one for the ad slots, one for the newsletter, three for SEO. Each one is maintained by a different party on a different release cadence. The site works, but nobody can predict what a routine update will break.

Performance under spikes. A general CMS with fifteen plugins executes a lot of code to render a page that is, semantically, one article. For a news site where a single story can bring 40× normal traffic in ten minutes, that overhead stops being academic.

Editorial friction. This is the one clients articulate least well and feel most. If publishing a story with a photo gallery, a source credit, a related-articles block, and an agency attribution takes eleven clicks across four screens, your newsroom will invent workarounds, and your data model will slowly fill with those workarounds.

Cost of ownership. Ten commercial add-on licences plus a maintenance retainer, renewed annually, forever.

Write your trigger down. If it's performance, you need before/after numbers. If it's editorial friction, you need to time the actual publishing workflow before and after. A migration that can't be measured will be judged on vibes, and vibes after a migration are always bad for the first two weeks.

The part everyone underestimates: URLs

Here is the single highest-risk element of any migration of an established site: the URL is not a detail, it's the asset.

A news site that has been publishing for eight years has tens of thousands of indexed URLs. Some carry external links. Some rank on page one for terms the business depends on. Every one of them that returns a 404 after cutover is value thrown away, and search engines are considerably slower to give it back than they were to take it.

So before touching anything else:

Export the complete URL inventory. Not the sitemap — the sitemap is what the old CMS thinks exists. Pull the real list from server access logs (every distinct path with a 200 in the last 12 months), from the search console's indexed-pages report, and from the database. Union all three. The gap between the sitemap and the logs is routinely 20–30%, and it's where the old, still-ranking content lives.
Classify it. Articles, category pages, tag pages, author pages, static pages, media files, feeds, pagination, and the long tail of oddities — print views, AMP variants, tracking-parameter duplicates, dated archive pages.
Decide the target URL scheme once, on purpose. If the new system can preserve the existing pattern, preserve it. Redesigning URLs "while we're in there" doubles the risk of the migration for aesthetic gain. If you genuinely must change the scheme, you now owe every old URL a permanent redirect.
Build the redirect map as data, not as rules. Regex rules feel elegant and fail silently on the exceptions. A generated lookup table of old path → new path, produced from the actual inventory, is boring and verifiable. You can test all 40,000 entries automatically; you cannot test a regex against the URLs you forgot existed.
Verify with a crawl, not a spot check. After cutover, crawl the full old inventory against the live site and assert: status is 301 (not 302, not 200-with-different-content), the destination resolves 200 in one hop, and no chains or loops exist. Redirect chains are where migrations quietly leak ranking.

The rule I'd tattoo on a junior developer: a migration is not done when the content is moved; it's done when the old URLs still work.

Content models never map one-to-one

The second discovery of every migration is that the old content model is not what the documentation says. It's what editors made it.

You will find, reliably:

A "category" that is actually used as a workflow flag ("Ready", "Legal check").
Three tags that mean the same thing with different capitalisation, plus one with a trailing space.
HTML pasted into a plain-text summary field, because the summary needed a line break in 2019.
Images referenced by absolute URL to the old domain, inside article bodies.
A custom field that is empty on 90% of records and load-bearing on the other 10%.

Two practical responses. First, profile before you map. For every field you intend to migrate, run the actual distribution: how many distinct values, how many nulls, the longest value, how many contain HTML, how many contain absolute URLs. Ten minutes of profiling prevents a mapping decision built on an assumption.

Second, decide explicitly what not to migrate. Migration is the one moment when dropping dead weight is cheap. Fifteen years of unused tags, an abandoned events module, a decade of spam comments — carrying them forward means maintaining them forever. Just make the decision consciously and record it, rather than discovering the loss in month three.

The transformation itself I'd keep as a repeatable, re-runnable script, never a hand-edited dump:

read source record
  → normalise (trim, fix encoding, resolve relative media paths)
  → map fields to target model
  → validate (required fields present, slug unique, dates sane)
  → write, or write to a rejects file with the reason

Re-runnable matters more than it sounds. You will run the migration many times — against a copy, into staging, into a rehearsal, and finally for real. If the process is a script that takes the source and produces the target from scratch, every run is identical and every fix is permanent. If it's a script plus "and then I manually fixed those 40 rows", you have no repeatable process, and the final run will differ from the rehearsal in ways you can't see.

Media is where the disk space and the surprises are

Content rows are small. Media is not, and it's messier:

Files referenced in article bodies by absolute URL — these break the moment the domain or path changes and must be rewritten during transformation.
Orphan files: uploaded, never referenced. Often 30–50% of the media directory.
Referenced files that don't exist. Someone cleaned up a folder in 2021.
Generated thumbnail variants in a naming scheme the new system doesn't share.
Filenames with non-ASCII characters and spaces, which behave differently across filesystems and web servers.

Before the migration, produce two lists: referenced-but-missing (fix or accept — these become broken images), and present-but-unreferenced (usually skip). Never regenerate thumbnails on first request under production traffic; pre-generate them during the migration, or your launch-day load test will be an unintentional one.

Rehearse the cutover, then rehearse it again

The cutover is a sequence of steps under time pressure, which is exactly the condition in which people improvise. So write it down as a runbook with times, owners, and — for each step — the answer to "how do we undo this?"

A workable shape:

T-7 days: full migration into staging with production data. Editors do real work in it for a week. This is the only reliable way to surface editorial-workflow problems, and it's the step most often skipped for schedule reasons.
T-2 days: dry run of the entire cutover against staging, stopwatch running. You now know how long the real one takes. It's always longer than the estimate.
T-1 day: DNS TTL lowered. Content freeze agreed with the newsroom, in writing, with a named person responsible.
T-0: final delta sync (only content created since the main import), switch, smoke tests, redirect crawl.
T+1 hour: the checks below.
T+7 days: daily monitoring of crawl errors and index coverage.

Pick your window by traffic, not by convention. For a news site the quiet hour is usually early morning local time — but check the actual analytics, and check the editorial calendar. Cutting over the night before a major scheduled news event is a self-inflicted wound.

Define the rollback before you need it. What's the trigger (error rate above X, homepage down more than Y minutes)? Who decides? How long does reverting take? A rollback plan that takes four hours isn't a rollback plan; it's a wish. In practice this usually means: keep the old stack running and untouched, switch at the DNS or proxy layer, and don't decommission anything for at least a fortnight.

What to check in the first hour, and the first month

Immediately after cutover, in this order:

Homepage and a sample of article pages return 200 with the right content.
The redirect crawl passes across the full old-URL inventory.
Feeds and the sitemap are valid and reachable; submit the new sitemap.
robots.txt is the production one. (Staging's Disallow: / reaching production is a classic, and it is genuinely expensive.)
Canonical tags and structured data render correctly on article pages.
Search: does querying an old, well-known headline return it?
Publishing works end to end — an editor publishes a real story with an image while you watch.
Analytics and any ad slots are firing.

Then over the following weeks, watch crawl errors (a slow rise in 404s means the URL inventory had gaps), index coverage (a drop means something is deindexing content — check canonicals and robots directives first), server response time under real traffic, and the numbers that justified the migration in the first place.

Expect a short dip in traffic. A one-to-two week wobble while search engines re-crawl is normal even on a clean migration. A sustained four-week decline is not a wobble; it's a bug, and it's nearly always URLs, canonicals, or robots directives — in that order of likelihood.

The things I'd tell myself before the first one

Preserve URLs. Everything else is negotiable.
Profile the real data before mapping it. The schema is a claim; the data is the fact.
Make the migration a re-runnable script, never a manual fix-up.
Rehearse with editors, not just with developers. They will find in one afternoon what a test suite won't find in a month.
Keep the old system alive and reversible for two weeks. It costs a little hosting and buys a lot of sleep.
Measure the thing you migrated for. Otherwise the only feedback you'll get is the first complaint.

None of this is exotic. A CMS migration is mostly the discipline to do unglamorous inventory work before the interesting part, and the honesty to define in advance what "it went wrong" looks like.

If you've done a large migration, I'm curious what bit you that isn't on this list — the failure modes seem to be endlessly inventive.

PHP 8.2 in 2026: Why It's Still the Best Choice for News & E-commerce CMS

Mahmut Gündüzalp — Thu, 02 Jul 2026 13:16:52 +0000

Every year someone announces that PHP is finished, and every year a large slice of the web keeps running on it — including most of the news portals and online stores I work with. That's not nostalgia. After building content and commerce systems for 200+ production sites, PHP 8.2 keeps winning the specific fight that matters for this domain: shipping a stable, fast, server-rendered site that a small team can maintain for years without a rewrite.

This isn't a "PHP is actually cool now" hype piece. It's the concrete reasons a boring, typed, request-per-page runtime is still the right default for news and e-commerce CMS work in 2026 — and the honest places where it isn't.

The workload, not the benchmark

News and commerce sites have a very particular shape:

Read-heavy, cache-friendly. Most visitors are anonymous and see the same article or product page. You want to render HTML on the server, cache it hard, and get out of the way.
Spiky. A breaking story or a campaign can multiply traffic in minutes. The runtime has to degrade gracefully, not fall over.
Long-lived. These sites run for 5–10 years. The team that maintains them in year 6 is rarely the team that built them in year 1.
SEO-critical. For news, Google News and fast Largest Contentful Paint aren't nice-to-haves; they're the business.

PHP's execution model fits this shape almost embarrassingly well. Each request starts clean, does its work, and dies. No long-lived process accumulating memory leaks, no shared mutable state to reason about across requests. For a page that reads from a database, renders a template, and returns HTML, "shared-nothing" is a feature, not a limitation.

What 8.2 actually gave us

The jump from PHP 5.x/7.x thinking to 8.2 changed how these codebases read. A few features carry most of the weight in practice.

readonly properties made value objects trustworthy. In a CMS you pass a lot of small immutable things around — a resolved article, a price with currency, a category node. Being able to say "this cannot change after construction" at the language level removes a whole category of "who mutated this?" bugs.

final class Money
{
    public function __construct(
        public readonly int $amount,      // minor units (kuruş/cents)
        public readonly string $currency, // 'TRY', 'USD'
    ) {}

    public function withVat(int $ratePercent): self
    {
        return new self(
            (int) round($this->amount * (100 + $ratePercent) / 100),
            $this->currency,
        );
    }
}

No setter, no accidental mutation three layers down, and money math stays in integer minor units where it belongs.

Enums (from 8.1, but they land fully in 8.2 codebases) replaced the pile of const STATUS_DRAFT = 0 integers that every legacy CMS drags around. An article status or an order state becomes a real type the IDE and the type checker understand:

enum OrderStatus: string
{
    case Pending  = 'pending';
    case Paid     = 'paid';
    case Shipped  = 'shipped';
    case Refunded = 'refunded';

    public function isFinal(): bool
    {
        return $this === self::Refunded || $this === self::Shipped;
    }
}

Constructor promotion + typed properties cut the ceremony that made older PHP feel heavy. A service class is now mostly signal:

final class ArticleRenderer
{
    public function __construct(
        private readonly TemplateEngine $view,
        private readonly CacheInterface $cache,
    ) {}
}

never return types, readonly classes, and stricter type coercion together mean the type checker catches far more before code ever reaches staging. On a long-lived codebase, that's the difference between refactoring with confidence and refactoring with prayer.

None of these are flashy. Collectively they turn PHP from "scripting language you tolerate" into "typed application language that happens to have the best deployment story on the web."

Server-rendered HTML is a competitive advantage again

For a few years the default answer to "how do I build the frontend?" was a JavaScript SPA. For a news article or a product page, that was almost always the wrong trade. You paid a bundle-size and complexity tax to re-implement, in the browser, the one thing the server already does perfectly: turn data into HTML.

PHP renders HTML on the server as its native act. Pair it with a template engine and a sprinkle of hypermedia-style JavaScript for the genuinely interactive bits (filters, infinite scroll, cart updates) and you get:

First paint that doesn't wait on a JS runtime — good for LCP, good for Google News.
HTML that's fully present for crawlers and AI scrapers without a headless-render step.
A frontend a backend developer can maintain, instead of a second full stack.

The industry rediscovering server-rendered HTML in 2024–2026 was, from a PHP seat, watching everyone walk back to where the language already stood.

Handling the spikes: cache, don't scale

The read-heavy, spiky profile has a well-worn answer in PHP land, and 8.2's execution model makes it clean:

Full-page cache for anonymous traffic. Most visitors on a breaking story are logged out and identical; serve them a cached HTML page and never touch the database.
Opcode + preloading so the framework itself isn't re-parsed on every request.
A cache layer that switches on under load rather than being always-on — normal traffic hits the database for freshness, a spike flips the site into aggressive caching automatically.

Because each request is isolated, there's no warm-up state to protect and no cross-request corruption to worry about when you throw a cache in front. The mental model stays simple, which is exactly what you want at 3 a.m. when a story goes viral.

The maintenance argument nobody puts on slides

The feature that keeps me choosing PHP 8.2 for client work isn't in any release note: you can hand the codebase to a different developer in year 4 and they can be productive in a week.

The request lifecycle is obvious. Request in, response out.
Deployment is rsync and a cache flush, not an orchestration diagram.
The hosting is everywhere and cheap, which matters enormously for regional news sites and small stores.
The type system now documents intent well enough that a newcomer can read a service class and know what it does.

For software that has to outlive its authors, that boringness is the whole point. A clever runtime that only its original author understands is a liability on a 10-year site.

Where PHP 8.2 is not the answer

Being honest keeps this credible. I don't reach for PHP when:

The workload is long-lived and stateful — a websocket server, a real-time collaboration backend, a streaming pipeline. Shared-nothing-per-request is the wrong shape there; that's a job for a long-running process in another runtime.
The product is fundamentally a rich client app — a design tool, an editor with heavy live interaction. That genuinely wants a real frontend stack.
You need CPU-bound number crunching. PHP will do it; it won't be the tool you're happy with.

News and e-commerce CMS work is none of those. It's data-in, HTML-out, cache-heavy, maintained-for-years. That's PHP's home field.

The 2026 verdict

Language choice is a maintenance decision disguised as a technical one. For content and commerce systems, the constraints that actually bite are: fast server-rendered pages, graceful behavior under spikes, cheap ubiquitous hosting, and a codebase a small team can still understand years later. PHP 8.2 — typed, readonly, enum-shaped, opcode-cached — hits every one of those without asking you to be clever.

That's why the sites I build for news and e-commerce still start from PHP 8.2, and why I expect them to still be running, and still be maintainable, long after the next "PHP is dead" post. If you want to see what a modern PHP news and e-commerce stack looks like in production, that's the whole business at alestaweb.com.

Pick the runtime you can still afford to maintain in year six. For this domain, that's still PHP.

Building Newsroom AI Modules in PHP: 50+ Specialized Workflows

Mahmut Gündüzalp — Fri, 26 Jun 2026 08:50:23 +0000

When people picture "AI in a newsroom," they usually imagine one big chat box bolted onto the editor. That's the wrong mental model, and it's the reason most of those features get used twice and then ignored.

A newsroom doesn't have one AI need. It has dozens of small, specific, repetitive ones: write three headline options, pull a 280-character social blurb, suggest a category, tag entities, generate alt text for the lead image, flag a defamation risk, translate the dek into English, propose an SEO title under 60 characters. Each is a tiny job with its own input shape, its own output contract, and its own tolerance for being wrong.

After running this across 200+ news sites, the architecture that actually stuck was not "an AI feature." It was a registry of small, specialized workflows behind a uniform interface. This article is how that's built in PHP, and the handful of decisions that matter more than which model you pick.

The unit of work: a task, not a prompt

The first thing to get right is the boundary. The reusable unit is a task — a named workflow with a fixed contract — not a free-floating prompt string. A task knows three things: what it needs, what it promises to return, and how expensive it's allowed to be.

interface AiTask
{
    public function name(): string;          // 'headline.suggest'
    public function tier(): string;          // 'cheap' | 'standard' | 'premium'
    public function build(array $input): array;   // -> messages for the model
    public function parse(string $raw): array;    // -> validated structured output
}

That parse() method is the part teams skip and regret. A model that returns prose when you expected JSON is not an edge case — it's Tuesday. If every task is responsible for validating its own output, a bad response fails inside the task where you can retry or fall back, instead of leaking malformed data into the editor.

A concrete task looks like this:

final class HeadlineSuggestTask implements AiTask
{
    public function name(): string { return 'headline.suggest'; }
    public function tier(): string { return 'cheap'; }

    public function build(array $input): array
    {
        return [
            ['role' => 'system', 'content' =>
                'You are a Turkish news copy editor. Return exactly 3 headline '
                . 'options as a JSON array of strings. Max 70 characters each. '
                . 'No clickbait, no ALL CAPS.'],
            ['role' => 'user', 'content' => mb_substr($input['body'], 0, 4000)],
        ];
    }

    public function parse(string $raw): array
    {
        $data = json_decode($this->stripFences($raw), true);
        if (!is_array($data) || count($data) < 1) {
            throw new BadOutputException('headline.suggest: not a list');
        }
        return array_slice(array_map('strval', $data), 0, 3);
    }
}

Once you have one of these, you have the shape for fifty. The interesting work isn't writing the fiftieth task — it's the machinery around them.

Why "50+" is an architecture choice, not a brag

The number isn't the point; the granularity is. You could collapse "suggest headline," "suggest SEO title," and "suggest social blurb" into one mega-prompt that returns all three. Don't. Three reasons:

Different cost tiers. A category suggestion can run on a fast, cheap model. A legal-risk flag should not.
Different failure handling. If headline generation fails, you shrug and the editor types one. If entity tagging fails silently, your archive search quietly rots.
Different UX surfaces. The blurb belongs to the social scheduler; the alt text belongs to the image picker. Coupling them in one call couples two unrelated screens.

Small tasks compose. Big prompts calcify.

Here's the rough taxonomy that emerged — grouped, because grouping is how editors actually find them:

Group	Example tasks
Headlines & framing	headline options, SEO title, social blurb, push-notification text
Structure	summary/dek, key-points list, "read more" suggestions
Classification	category suggest, tag/entity extraction, topic clustering
Media	image alt text, caption draft, thumbnail crop hint
Quality & risk	tone check, defamation/risk flag, fact-claim highlighter, profanity filter
SEO & distribution	meta description, schema keywords, related-article linking
Language	translate dek, simplify, localize idiom

That's already 25+ before you count per-language and per-section variants. The registry is what keeps it from becoming chaos.

The registry and the router

The registry is boring on purpose: a name-to-task map. The router is where the one genuinely valuable idea lives — routing by tier, not by vibes.

final class AiRouter
{
    /** @param array<string, AiTask> $tasks */
    public function __construct(
        private array $tasks,
        private ModelGateway $gateway,
        private AiCache $cache,
    ) {}

    public function run(string $taskName, array $input): array
    {
        $task = $this->tasks[$taskName]
            ?? throw new UnknownTaskException($taskName);

        $key = $this->cacheKey($taskName, $input);
        if ($hit = $this->cache->get($key)) {
            return $hit;
        }

        $messages = $task->build($input);
        $raw      = $this->gateway->complete($task->tier(), $messages);

        try {
            $out = $task->parse($raw);
        } catch (BadOutputException $e) {
            // one retry on the next tier up before giving up
            $raw = $this->gateway->complete($this->escalate($task->tier()), $messages);
            $out = $task->parse($raw);
        }

        $this->cache->put($key, $out, $task->ttl());
        return $out;
    }
}

Three things are doing real work here and each earns its place:

Caching by (task, input) hash. The same article body gets a headline suggestion once. Editors click these buttons repeatedly; without a cache you pay for every nervous re-click. This single layer was the biggest cost reduction we measured.
Tier-based escalation on bad output. Cheap model first. If it returns garbage that fails parse(), retry once on a stronger tier. Most cheap-model failures are formatting failures, and they don't repeat on the better model. You get cheap-model economics with premium-model reliability on the tail.
The gateway hides the provider. complete(tier, messages) is the entire surface the task sees. Whether 'cheap' maps to one provider this month and another next month is an ops decision, not a code change in 50 tasks.

The gateway: one seam for every provider

The gateway is what makes provider diversity survivable. News work is bursty and rate limits are real, so you want the freedom to move a tier between providers without touching task code.

final class ModelGateway
{
    public function __construct(private array $tierConfig) {}

    public function complete(string $tier, array $messages): string
    {
        $cfg = $this->tierConfig[$tier];          // provider + model + limits
        $provider = ProviderFactory::make($cfg['provider']);

        return $provider->chat(
            model:    $cfg['model'],
            messages: $messages,
            maxTokens: $cfg['max_tokens'],
        );
    }
}

The payoff is operational, not architectural elegance for its own sake. When a provider degrades at 9 a.m. on an election morning — and it will — you change a config map, not a deployment. Keeping the model identity out of the task and in the tier config is the difference between a five-minute mitigation and a panic.

Quality gates: cheap models lie confidently

A specialized-task design tempts you toward cheap models everywhere, because each job is small. The trap is that small jobs still produce confidently wrong output. The defense is deterministic gates around non-deterministic output — code, not another model, checks the boring constraints.

function validateHeadlines(array $headlines): array
{
    return array_values(array_filter($headlines, function (string $h): bool {
        $len = mb_strlen($h);
        if ($len < 15 || $len > 70)           return false; // length contract
        if ($h === mb_strtoupper($h))          return false; // no ALL CAPS
        if (preg_match('/!{2,}|\?{2,}/', $h))  return false; // no "!!!"
        return true;
    }));
}

Notice there's no model in that function. The expensive judgment ("is this a good headline?") stays with the human editor. The cheap, mechanical judgment ("is this even a valid headline-shaped string?") is plain PHP that runs in microseconds and never hallucinates. Push every constraint you can express as code out of the prompt and into a gate. Prompts are for taste; code is for rules.

The one place to spend a premium model deliberately is risk — defamation, sensitive claims, anything where a wrong call has legal weight. That task should run on your strongest tier, never cache its "looks fine" verdict for long, and always present as advisory to a human. AI flags; people decide.

Asynchronous by default

Editors will not wait four seconds for a button. Anything slower than roughly a second belongs in the background, with the result arriving when it's ready rather than blocking the save.

// On save: enqueue, don't block.
$queue->push('ai.enrich', [
    'article_id' => $id,
    'tasks'      => ['summary.make', 'tag.extract', 'category.suggest', 'seo.meta'],
]);

// A worker drains the queue and writes suggestions back as drafts
// the editor can accept or ignore — never auto-published.

Two rules that saved us real pain:

Suggestions are drafts, never silent writes. AI output lands in a "suggested" state. A human accepts it. The day you let a model write directly to the published field is the day you explain a hallucinated dateline to your editor-in-chief.
Idempotent jobs. Queues retry. If summary.make runs twice, the second run should overwrite the same suggestion slot, not create a duplicate. Key the write by (article_id, task_name).

What I'd tell my past self

Model the task contract first, the prompt second. The interface outlives any specific model.
Validate output inside the task. Malformed responses are normal; treat them as control flow, not exceptions to your worldview.
Route by tier, cache by input. These two together did more for cost and reliability than any prompt-engineering cleverness.
Keep rules in code, taste in prompts. Every constraint you can check deterministically is one the model can't violate.
Async, advisory, idempotent. The newsroom trusts a tool that suggests and never surprises.

The "50+" isn't a feature count to put on a slide. It's what falls out naturally once each AI job is small enough to have a single clear contract. Build the seam — task, registry, router, gateway, gate — and adding the fifty-first workflow is an afternoon, not a project.

We've been refining this pattern in production news software at Alesta WEB, across publishers of very different sizes, and the lesson keeps repeating: the architecture, not the model, is what makes newsroom AI feel reliable instead of gimmicky.

KVKK, İYS, BİK: Turkish Software Compliance for Engineers (with PHP examples)

Mahmut Gündüzalp — Sun, 21 Jun 2026 12:45:03 +0000

If you build software that serves Turkish users, three acronyms will eventually land on your desk: KVKK, İYS, and BİK. They are not optional "nice to have" features — they are legal obligations with real fines attached. Yet most engineering write-ups about them are written by lawyers, for lawyers, and stop exactly where the interesting part begins: the code.

This is the article I wish I had when we first had to make 200+ production sites compliant. It's the engineer's view — what each rule actually requires from your application, and how to implement it cleanly in PHP. The examples are intentionally generic; adapt the storage and framework details to your own stack.

The three you'll meet

Acronym	Full name	What it governs	Who enforces it
KVKK	Kişisel Verilerin Korunması Kanunu	Personal data protection (Turkey's GDPR analogue)	KVKK Authority (KVKK Kurumu)
İYS	İleti Yönetim Sistemi	Commercial electronic messages (SMS/email/calls)	Managed via a central national registry
BİK	Basın İlan Kurumu	Press/news site requirements & official announcements	Basın İlan Kurumu (for news publishers)

KVKK touches almost every app. İYS hits you the moment you send a marketing message. BİK is specific to news publishers. Let's take them in order.

1. KVKK — personal data protection

KVKK shares its DNA with GDPR, so if you've done GDPR work the concepts will feel familiar: lawful basis, explicit consent, data minimization, the right to erasure, breach notification, and registration with a central inventory (VERBİS) once you cross certain thresholds.

The mistakes I see most often are not legal misreadings — they're engineering shortcuts. Three of them matter.

Separate consent from the action

A checkbox that says "I accept the terms and consent to marketing" bundles two different lawful bases. KVKK wants explicit, specific, unbundled consent. In practice that means storing each consent as its own record with enough metadata to prove it later.

// Each consent is its own row — never a single boolean on the user.
function recordConsent(int $userId, string $purpose, bool $granted): void
{
    $stmt = $pdo->prepare(
        'INSERT INTO consent_log
            (user_id, purpose, granted, ip, user_agent, created_at)
         VALUES (:uid, :purpose, :granted, :ip, :ua, :now)'
    );
    $stmt->execute([
        'uid'     => $userId,
        'purpose' => $purpose,                 // e.g. 'marketing_email', 'analytics'
        'granted' => $granted ? 1 : 0,
        'ip'      => $_SERVER['REMOTE_ADDR'] ?? '',
        'ua'      => substr($_SERVER['HTTP_USER_AGENT'] ?? '', 0, 255),
        'now'     => date('Y-m-d H:i:s'),
    ]);
}

The point of the ip, user_agent, and timestamp is not surveillance — it's that when someone asks "did this user actually consent, and when?", you can answer with evidence instead of a shrug. Consent is also revocable, so you append a new granted = 0 row rather than mutating the old one. The history is the proof.

The cookie banner has to actually block scripts

A banner that loads Google Analytics and the Meta pixel before the user clicks "accept" is theatre. Under KVKK (as under GDPR) non-essential trackers must not fire until consent exists. The cleanest implementation is to gate the script tags server-side:

function trackingScripts(int $userId): string
{
    if (!hasConsent($userId, 'analytics')) {
        return ''; // nothing loads, no third-party calls
    }
    return '<script src="/js/analytics.js" defer></script>';
}

If your analytics is purely first-party and aggregated (no cross-site identifiers, no PII), you have a much easier compliance story — which is one practical reason a lot of Turkish products lean toward self-hosted, first-party analytics.

Erasure means erasure (but keep the legal minimum)

The right to be forgotten collides with other obligations: you may be legally required to keep invoice and transaction records for years. The resolution is anonymization, not blind deletion. Strip the identifying fields, keep the financially/legally required skeleton.

function eraseUser(int $userId): void
{
    // Anonymize, don't orphan: invoices must survive for tax law.
    $pdo->prepare(
        "UPDATE users SET
            name='[deleted]', email=CONCAT('deleted_', id, '@invalid'),
            phone=NULL, address=NULL, anonymized_at=:now
         WHERE id=:id"
    )->execute(['now' => date('Y-m-d H:i:s'), 'id' => $userId]);

    // Hard-delete things with no retention requirement.
    $pdo->prepare('DELETE FROM consent_log WHERE user_id=:id')
        ->execute(['id' => $userId]);
}

Decide your retention periods deliberately, document them, and make them queryable. "We delete logs after N days" is a sentence you want to back with a cron job, not a hope.

2. İYS — the commercial message registry

İYS is the part that surprises foreign engineers. In Turkey there is a central national registry for commercial electronic message consent. Before you send a marketing SMS, email, or call to someone, their consent must exist in that registry — and your own database agreeing isn't enough on its own.

The engineering consequence: never send marketing straight from your app's consent table. You check İYS first. Transactional messages (order confirmations, password resets, shipping updates) are exempt — but the line between "transactional" and "marketing" is exactly where people get fined, so be conservative.

function canSendMarketing(string $recipient, string $channel): bool
{
    // 'channel' is one of: MESAJ (SMS), EPOSTA (email), ARAMA (call)
    $status = iysLookup($recipient, $channel); // calls the registry/integrator API

    // Only an explicit, active opt-in counts.
    return $status === 'ONAY';   // vs 'RET' (rejected) or 'ALICI_YOK' (unknown)
}

function sendCampaign(array $recipients, string $channel, string $message): void
{
    foreach ($recipients as $r) {
        if (!canSendMarketing($r, $channel)) {
            logSkipped($r, 'no_iys_consent');
            continue;            // skipping is cheaper than a fine
        }
        dispatch($channel, $r, $message);
    }
}

Two practical notes from running this at scale:

Sync both directions. When a user opts in or out in your UI, push that change up to the registry. When the registry changes (a user opts out there), pull it down. A nightly reconciliation job catches drift.
Cache lookups, but not forever. Hammering the lookup API per-recipient on a large campaign is slow and rude. A short-TTL cache (hours, not weeks) keeps you fast without letting stale "ONAY" values send messages to someone who opted out this morning.

3. BİK — for news publishers

If you don't run a news site, skip this section. If you do, BİK compliance is what stands between you and being eligible for official announcements (resmî ilan) and the credibility that comes with it.

BİK's requirements are more editorial than algorithmic, but several do translate into application features:

Mandatory imprint (künye): publisher identity, responsible editor, contact, and address must be present and reachable. This is a structured page, not a footer afterthought.
Source attribution (mahreç): agency-sourced content must be labelled with its origin. If you ingest from news agencies, carry the source field through your pipeline to the rendered article.
Content cadence & originality: a minimum flow of original editorial content, which in practice means your CMS needs solid authorship, scheduling, and audit metadata.
Archive integrity: published pieces should remain accessible and stable at their URLs.

The imprint and source-attribution pieces are the ones that bite teams late, because they're easy to bolt on at the end and easy to get subtly wrong. Model them as first-class fields from the start:

// Carry the source through to render — don't lose it in the import step.
function renderArticleMeta(array $article): string
{
    $out = '<div class="article-meta">';
    $out .= '<span class="author">' . e($article['author_name']) . '</span>';
    if (!empty($article['source_agency'])) {
        $out .= '<span class="source">Kaynak: '
              . e($article['source_agency']) . '</span>'; // mahreç
    }
    $out .= '<time datetime="' . e($article['published_at']) . '">'
          . e(formatTr($article['published_at'])) . '</time>';
    return $out . '</div>';
}

This is the kind of requirement that pushed our news CMS work toward treating source, author, and publish metadata as non-negotiable columns rather than optional extras — it's far cheaper than retrofitting attribution across an archive later.

A checklist you can paste into a ticket

[ ] KVKK: consent stored per-purpose, with timestamp + evidence, revocable as an append
[ ] KVKK: non-essential trackers gated server-side, fire only after consent
[ ] KVKK: erasure = anonymize-and-retain-legal-minimum, with documented retention periods
[ ] KVKK: VERBİS registration checked against your data-controller thresholds
[ ] İYS: registry lookup before every marketing send; transactional messages clearly separated
[ ] İYS: two-way sync (UI ⇄ registry) + nightly reconciliation + short-TTL cache
[ ] BİK: structured imprint (künye) page, reachable
[ ] BİK: source attribution (mahreç) carried end-to-end from import to render
[ ] BİK: stable article URLs and accessible archive

Closing thought

None of this is exotic engineering. It's mostly the discipline of modeling consent and provenance as data you can prove, not state you assume. Do it at the schema level on day one and compliance becomes a property of the system. Bolt it on at the end and it becomes a migration you'll dread.

We've shipped this pattern across 200+ Turkish production sites at Alesta WEB, and the single biggest lesson is the boring one: make consent and source first-class data, and the legal requirements mostly take care of themselves.

This is general engineering guidance, not legal advice — confirm specifics for your situation with a qualified professional.

How We Reduced LLM Costs by 95%: Cache + Batch + Cascade in PHP

Mahmut Gündüzalp — Sat, 13 Jun 2026 11:10:36 +0000

How We Reduced LLM Costs by 95%: Cache + Batch + Cascade in PHP

We build news software — a content platform used by more than 200 publishers at Alesta WEB. Once we wired language models into the newsroom workflow (headline suggestions, summaries, SEO fields, tag extraction, draft scaffolding), something predictable happened: the AI bill started growing faster than the feature list.

The naive version of "add AI" is a thin wrapper around one expensive frontier model, called fresh on every request. It works in a demo. In production, across thousands of articles a day, it's a slow way to set money on fire.

This is the architecture we settled on after eighteen months of running it. Three layers — cache, batch, cascade — plus the quality gates that make the cheap layers safe to rely on. The result was roughly a 95% reduction in per-task cost versus the naive "frontier-model-only, no cache" baseline, with no measurable drop in editorial quality.

The code is PHP, because the platform is PHP. The ideas are language-agnostic.

1. The Naive Approach (and What It Costs)

Here's the version almost everyone ships first:

function generateHeadline(string $articleBody): string
{
    $response = $client->chat([
        'model'    => 'gpt-4o',
        'messages' => [
            ['role' => 'system', 'content' => 'You write concise news headlines.'],
            ['role' => 'user',   'content' => $articleBody],
        ],
    ]);

    return $response['choices'][0]['message']['content'];
}

Nothing is wrong with this code. The problem is the usage pattern around it:

The same wire-service story gets posted by dozens of sites, so we generate near-identical headlines over and over.
Editors regenerate three or four times to compare options.
A frontier model is doing work — extracting tags, normalizing a category — that a model costing a fraction as much would do just as well.
Every call is synchronous and real-time, even when nothing about the task is urgent.

When you multiply "fresh frontier call every time" by real newsroom volume, the per-article AI cost lands somewhere that makes the CFO ask uncomfortable questions. Each of the three layers below attacks one of those waste sources.

2. Layer 1 — Cache: What to Cache, and What Not To

The single biggest win is the most boring one: don't ask the same question twice.

A large share of LLM calls in a news system are functionally identical. The key insight is that the cache key should be derived from the meaningful inputs — the task type, the model, the prompt template version, and a normalized hash of the content — not from the raw request object.

function cachedComplete(string $task, string $content, callable $compute): string
{
    $key = sprintf(
        'llm:%s:%s:%s',
        $task,
        PROMPT_VERSION[$task],          // bump to invalidate on prompt change
        hash('xxh128', normalize($content))
    );

    $hit = $cache->get($key);
    if ($hit !== null) {
        return $hit;                    // ~0 cost, sub-millisecond
    }

    $result = $compute($content);       // the actual API call
    $cache->set($key, $result, ttlFor($task));
    return $result;
}

function normalize(string $s): string
{
    // collapse whitespace, strip volatile boilerplate so trivially
    // different inputs map to the same key
    return trim(preg_replace('/\s+/u', ' ', $s));
}

Two details matter more than the cache engine you pick (we use Redis, but a database table works):

Version the prompt in the key. When you change a prompt template, you want every cached answer for that task to become a miss. Putting a PROMPT_VERSION constant into the key turns prompt edits into a clean, instant invalidation instead of a stale-output bug you chase for a week.

Know what not to cache. Anything personalized, anything real-time, anything where two identical inputs should legitimately produce different outputs (a "give me a fresh alternative" button) must bypass the cache. We mark those tasks explicitly rather than relying on TTL alone.

In our workload the cache hit rate sits around 60–70%, mostly because syndicated content repeats across sites. That one layer alone removes well over half the spend.

3. Layer 2 — Batch APIs: Trade Latency for Money

A surprising amount of LLM work in a newsroom is not time-sensitive. Nightly re-tagging of the archive. Generating summaries for the previous day's articles. Backfilling SEO descriptions on older content. None of it needs an answer in 800 milliseconds.

The major providers offer batch endpoints that run asynchronously — you submit a file of requests, and within a window (typically up to 24 hours) you get the results back, at roughly half the price of the synchronous API.

// Collect non-urgent jobs into a single batch submission
$lines = [];
foreach ($pendingJobs as $job) {
    $lines[] = json_encode([
        'custom_id' => $job->id,
        'method'    => 'POST',
        'url'       => '/v1/chat/completions',
        'body'      => [
            'model'    => $job->model,
            'messages' => $job->messages(),
        ],
    ]);
}

$batchFile = uploadJsonl(implode("\n", $lines));
$batch     = $client->batches()->create([
    'input_file_id'     => $batchFile['id'],
    'endpoint'          => '/v1/chat/completions',
    'completion_window' => '24h',
]);

// A worker polls for completion and writes results back by custom_id
$queue->later('poll_batch', ['batch_id' => $batch['id']], minutes: 15);

The discipline this forces is healthy: you have to classify each task as interactive (editor is waiting) or deferred (a cron job can handle it tonight). Once we did that audit, far more work turned out to be deferrable than we expected. Roughly a quarter of our remaining spend — after caching — moved onto batch pricing for a flat ~50% discount on that slice.

A caveat: batch pricing differs by provider, and so does the completion window and the failure behavior. Build your batch layer behind an interface so the provider is swappable, and always handle partial failures — a batch of 5,000 requests will occasionally return 4,997.

4. Layer 3 — Cascade Routing: Match the Model to the Task

The last layer is the one people resist, because it feels like cutting corners. It isn't — it's refusing to pay frontier prices for kindergarten work.

Not every task needs the smartest model. Extracting tags from a story, mapping a category, cleaning up whitespace, classifying sentiment — small, cheap models handle these perfectly. Reserve the expensive model for genuinely hard generation: nuanced summaries, editorial rewriting, anything where a mistake is visible to readers.

const TASK_TIER = [
    'tag_extraction'   => 'cheap',
    'category_map'     => 'cheap',
    'sentiment'        => 'cheap',
    'summary'          => 'mid',
    'headline'         => 'mid',
    'editorial_rewrite'=> 'frontier',
];

const TIER_MODEL = [
    'cheap'    => 'gpt-4o-mini',
    'mid'      => 'gpt-4o',
    'frontier' => 'the-strongest-model-you-trust',
];

function modelFor(string $task): string
{
    return TIER_MODEL[TASK_TIER[$task] ?? 'mid'];
}

We run six providers behind one interface (the editorial team never sees which one answered), which means cascade routing can also fail over: if a cheap model's output fails a quality gate, the task is automatically re-run one tier up. That gives you the cost of the cheap tier on the 90%+ of cases it handles well, and the safety of the expensive tier on the cases it doesn't.

Cascade routing is what takes you from "big savings" to "almost free for the easy majority."

5. Quality Gates: Keeping Cheap Models Honest

Cascade routing only works if you can detect when a cheap model got it wrong — otherwise you're trading money for garbage. Quality gates are cheap, deterministic checks that run on the output before it's accepted:

function passesGate(string $task, string $output): bool
{
    return match ($task) {
        'headline' => mb_strlen($output) <= 90
                      && !str_contains($output, "\n")
                      && !looksTruncated($output),

        'summary'  => mb_strlen($output) >= 100
                      && sentenceCount($output) >= 2,

        'tag_extraction' => isValidJsonArray($output),

        default    => $output !== '',
    };
}

None of these call an LLM. They're string length, format validity, structure checks — the kind of thing that costs nothing and catches the most common cheap-model failures (truncation, wrong format, empty output). When a gate fails, the cascade re-runs the task one tier up and logs it. If a particular task fails its gate too often, that's your signal to move it up a tier permanently.

This is the piece that makes the whole architecture trustworthy. Without gates, "use a cheaper model" is a gamble. With gates, it's a measured decision with an automatic safety net.

6. A Cost Dashboard You Actually Look At

You can't optimize what you don't measure. We log every LLM call with four fields: task, tier, whether it was a cache hit, and the token counts. That's enough to answer the only questions that matter:

Which task is costing the most? (Usually the surprise here is a "cheap" task being called a million times.)
What's our real cache hit rate, per task?
How often is the cascade escalating — and which tasks?

A weekly rollup of per-task economics turns cost control from a panic ("the bill doubled!") into a routine ("tag extraction escalation rate crept up, the prompt drifted, fix the template"). The dashboard is boring on purpose. Boring means in control.

7. The Numbers, After Eighteen Months

Against the naive baseline (one frontier model, every call fresh and synchronous), the layers compound:

Cache removes ~60–70% of calls outright.
Batch takes ~50% off a meaningful slice of what remains.
Cascade routes the easy majority of the rest to models costing a fraction of frontier prices.

Stacked, that lands at roughly 95% lower per-task cost for the same workload — and, because cache hits are instant and cheap models are fast, the median latency for AI features actually improved. Cheaper and faster, which is not the trade-off people expect when they hear "we cut the AI budget."

The editorial quality held because the expensive model still does all the work that's actually hard; we just stopped paying it to do the easy work.

8. What's Next: Prompt Caching

The newest lever we're rolling out is provider-side prompt caching — where a long, stable system prompt (style guide, formatting rules, examples) is cached on the provider's side and billed at a steep discount on repeat calls. For a news system with a large, rarely-changing editorial style prompt prepended to thousands of calls, that's a natural fit on top of the three layers above.

The throughline across all of it is the same: a language model is a power tool, not a default. Cache the repeats, defer what isn't urgent, route easy work to cheap models, and verify cheap output with checks that cost nothing. Do that, and AI features stop being a line item that scares the finance team and go back to being what they should be — a feature.

We build news software used by 200+ publishers — agency integration, AI-assisted editorial workflows, native mobile apps, and subscription infrastructure. More on the platform at alestaweb.com.

DEV Community: Mahmut Gündüzalp

The Socket Was Open. The Site Was Dead for Fourteen Hours.

What the checks reported

The log line that mattered

PPID 1

Why nothing restarted it

Fourteen hours

The fix, and it is boring

What to check instead

The short version

Lowercasing "İ" Adds a Character: How Turkish Text Broke a Keyword Router

The symptom

Trap 1: lowercasing a dotted capital I produces two code points

Unicode normalization will not save you

The other direction, for readers who do not write Turkish

Trap 2: unbounded substring matching

Trap 3: map order is a priority ranking you did not know you wrote

Normalize for matching, never for identity

How to verify, because you cannot eyeball this

The short version

The Word "null" Is Not a Null: How JSON Nulls Turn Into Text on Android

Why it does that

It also depends on which org.json you are compiled against

Where the null came from in the first place

Why the test accounts never caught it

Fix both layers, not one

Server side — normalize once, at the boundary

Client side — never call optString directly again

What about Gson, Moshi, kotlinx.serialization?

The five-minute audit

MySQL Will Quietly Eat Your Data If You Let It: Truncation, Charsets, and the 1366 You Never See

Part 1: The truncation

Finding the damage in a database you inherited

Part 2: Strict mode is necessary and not sufficient

Part 3: Charsets, where "it worked on the old server" lives

When you think data is corrupted, check the bytes

Part 4: The application-side version of the same bug

The checklist

Your Uptime Monitor Says 200. Your Contact Form Has Been Dead for Three Days.

What broke

Four ways to be broken and still return 200

What to assert instead

Check every route, not one route

Assert content, not just bytes

Diff the error log around every deploy

Round-trip the form

Why this keeps happening

The short version

The Query That Ate 75% of Our Database CPU: A MySQL Full-Text Post-Mortem

Symptom first

Find the query, not a query

The query

Why it's expensive

Fix 1: stop searching with a paragraph

Fix 2: rank in two stages

Results

Three things I learned that weren't the bug

What I'd take away

Self-Hosted MCP: Building a Model Context Protocol Server in PHP

What MCP actually is

The two transports

The handshake

What the PHP side looks like

Connecting it to Claude

The part I underestimated: exposure

Worth it?

From Generic CMS to Purpose-Built: What 200 Site Migrations Taught Me

Why sites migrate at all

The part everyone underestimates: URLs

Content models never map one-to-one

Media is where the disk space and the surprises are

Rehearse the cutover, then rehearse it again

What to check in the first hour, and the first month

The things I'd tell myself before the first one

PHP 8.2 in 2026: Why It's Still the Best Choice for News & E-commerce CMS

The workload, not the benchmark

What 8.2 actually gave us

Server-rendered HTML is a competitive advantage again

Handling the spikes: cache, don't scale

The maintenance argument nobody puts on slides

It also depends on which `org.json` you are compiled against

Client side — never call `optString` directly again