DEV Community: Mahmut Gündüzalp

Green Build, Broken Site: Three Next.js 16 Failures You Only See in Production

Mahmut Gündüzalp — Thu, 01 Oct 2026 12:20:21 +0000

One Next.js 16 site, one working day, three failures. None of them showed up on the development machine. All three passed the build. Each one broke a single feature quietly while the rest of the site looked fine.

Image uploads in the admin panel returned 422 for every file.
Files uploaded after the server started returned 404 until the next restart.
Scheduled posts went live in the database but never appeared on the site.

They share one root cause: the environment you test in is not the one you ship. Below is each failure, how it showed up, the fix, and the check that would have caught it before a user did.

1. The image library that compiled to nothing

Symptom

The admin upload endpoint re-encodes every image with sharp (strip metadata, resize, convert to WebP). In production it answered every request with:

{ "hata": "Görsel işlenemedi" }

That's "image could not be processed", the message from our own catch block. The interesting part was what the catch block was hiding.

Why it only happened in production

Locally we built with next build --webpack, because a font download failed under Turbopack on that Windows machine. The server ran a plain next build, which in Next.js 16 means Turbopack.

In the Turbopack output, the static import

import sharp from "sharp";

had been replaced with (void 0). The handler compiled, the route existed, and then the first call failed at runtime.

The usual advice is to list the package in serverExternalPackages. sharp is already on Next's built-in list, and adding it explicitly made no difference in that build.

Fix

Resolve the module at runtime, from the app's own node_modules, so that the bundler has nothing to analyse statically:

import { createRequire } from "node:module";
import path from "node:path";

type Sharp = typeof import("sharp");
const MODULE_NAME = "sharp"; // a variable, not a literal: nothing to resolve at build time
let cached: Sharp | null = null;

export function loadSharp(): Sharp {
  cached ??= createRequire(path.join(process.cwd(), "package.json"))(MODULE_NAME) as Sharp;
  return cached;
}

The type import keeps full typings, and loadSharp()(buffer).webp().toBuffer() behaves exactly as before.

The check that would have caught it

Build in CI with the same bundler you deploy with. If your local build needs --webpack to work around something, that is a second build pipeline, and only one of the two runs in production. The upload was tested, but against the webpack build.

2. Uploads that 404 until the next restart

Symptom

Once uploads worked, a new one appeared: the API returned a URL like /uploads/logos/abc123.webp, the file was on disk with correct permissions, and the URL returned 404. After a process restart the same URL worked.

Why

next start serves the public/ files it found when the process started. Anything written there afterwards is invisible to the static file handler: the same request goes from 404 to 200 after nothing but a restart. The dev server is more forgiving, which is why nobody noticed.

public/ is meant for assets that ship with the build, not for user content. Most guides tell you to move uploads to object storage, and that is the right long-term answer. But if you self-host on a single server and want to keep files on local disk, there is a small fix.

Fix

Keep writing to public/uploads/, and add a catch-all route handler that serves from that directory only when the static handler misses:

// app/uploads/[...path]/route.ts
import path from "node:path";
import { promises as fs } from "node:fs";

const ROOT = path.join(process.cwd(), "public", "uploads");
const TYPES: Record<string, string> = {
  ".webp": "image/webp", ".png": "image/png", ".jpg": "image/jpeg",
  ".jpeg": "image/jpeg", ".gif": "image/gif", ".avif": "image/avif",
};

export async function GET(_req: Request, { params }: { params: Promise<{ path: string[] }> }) {
  const { path: parts } = await params;
  if (!parts.length || parts.some((p) => !p || p === "." || p === ".." || p.includes("\\") || p.includes("\0"))) {
    return new Response("Not found", { status: 404 });
  }
  const file = path.resolve(ROOT, ...parts);
  const type = TYPES[path.extname(file).toLowerCase()];
  if (!type || !file.startsWith(ROOT + path.sep)) return new Response("Not found", { status: 404 });

  try {
    const st = await fs.stat(file);
    if (!st.isFile()) return new Response("Not found", { status: 404 });
    return new Response(new Uint8Array(await fs.readFile(file)), {
      headers: {
        "Content-Type": type,
        "Content-Length": String(st.size),
        "Cache-Control": "public, max-age=31536000, immutable", // names are random and never reused
        "X-Content-Type-Options": "nosniff",
      },
    });
  } catch {
    return new Response("Not found", { status: 404 });
  }
}

Three details matter more than the rest:

Path traversal. Reject .. segments and confirm that the resolved path is still under the root. Either check alone has gaps.
No SVG. An SVG can carry script. If you need SVG uploads, serve them as Content-Disposition: attachment or from another origin.
Immutable caching is only safe because file names are random. If your uploads keep their original names and can be overwritten, drop immutable.

Files present at startup are still served by the static handler. The route only handles the rest.

The check that would have caught it

Test upload-then-fetch against a production build, without restarting in between. "Upload returns 200" is not the test. "The returned URL serves the bytes" is.

3. Scheduled posts that published but never appeared

Symptom

A cron job calls an endpoint every ten minutes. The endpoint flips scheduled posts whose time has come to published, then invalidates the posts cache tag. The database showed the posts as published. The home page, the sidebar and the "latest posts" blocks still showed the old list, indefinitely.

The cron used curl -s ... > /dev/null, so its failures went nowhere. The process log had the answer:

Error: updateTag can only be called from within a Server Action. To invalidate
cache tags in Route Handlers or other contexts, use revalidateTag instead.

Why

Next.js 16 split tag invalidation in two:

updateTag(tag): Server Actions only. Read-your-own-writes semantics, so the user who submitted the form sees the change immediately.
revalidateTag(tag, profile): works anywhere on the server, including Route Handlers, and now takes a second argument.

The endpoint called updateTag. The database write happened first, then the call threw, and the cache was never touched. Because the post was already published, the next cron run found nothing to do, so it never retried. And because data cached with unstable_cache and no revalidate time stays until its tag is invalidated, "eventually" never came.

Fix

import { revalidateTag } from "next/cache";

// updateTag only works inside Server Actions; route handlers use revalidateTag.
if (result.count > 0) revalidateTag("posts", { expire: 0 });

{ expire: 0 } expires the entries immediately. The "max" profile would serve stale data once and refresh in the background. For "the post should be visible now", expire immediately.

We verified it end to end: inserted a scheduled test post, called the endpoint (a JSON body reporting one published post, instead of a 500), requested a page and saw the post in the sidebar, then deleted the test row.

One more trap: deleting a row with raw SQL invalidates nothing. After removing the test post, it stayed in the cached sidebar until the tag was invalidated again. Any out-of-band write (a migration, a manual fix, an import script) needs a tag invalidation afterwards, or a way to trigger one.

The check that would have caught it

Don't send cron output to /dev/null. Log the response body, or at least fail loudly on a non-2xx status (curl -fsS). The endpoint had been returning 500 for every run that had work to do.

What the three have in common

Failure	Dev	Prod	What actually differed
`sharp` was `undefined`	worked	422	webpack locally, Turbopack on the server
new uploads 404	worked	404 until restart	dev server re-scans `public/`, prod doesn't
cache never refreshed	not exercised	stale forever	the code path only ran from cron

None of these is a Next.js bug in the sense of "the framework is broken". Two are documented behaviours and one is a bundler edge case with a clean workaround. All three survived because the only environment that ran the code path was production, and production was set up to stay quiet about failures.

The checklist we now run before calling a deploy done:

Build with the production bundler, no flags that the server doesn't use.
Upload a file, then fetch the returned URL without restarting.
Run every cron endpoint once by hand and read the response body.
After any manual database change, invalidate the affected cache tags.

We build news, e-commerce and restaurant POS software at Alesta WEB. These notes come from moving our own site to a new stack this week.

Two Proxies, One Comma, Every Page a 500

Mahmut Gündüzalp — Mon, 28 Sep 2026 06:29:57 +0000

A Next.js site went live behind a CDN and every single page returned 500. Not the admin area, not the login route — the home page, the about page, static-looking marketing pages. The build was green. The same build ran without a single error on the development machine.

The cause was one header with one extra value in it:

x-forwarded-proto: https, https

This post walks through why that header looks the way it does, exactly where it breaks, why the obvious fix causes a second, different failure, and the two neighbouring headers that go wrong for the same reason.

All behaviour below was re-measured afterwards in isolation, against the same library build that was deployed. Nothing is from memory.

The shape of the deployment

browser ──https──▶ CDN ──https──▶ origin web server ──http──▶ Next.js (node)

Two reverse proxies sit in front of the application: the CDN at the edge, and the web server on the origin machine that terminates TLS and forwards to the Node process on a local port.

Both of them are doing their job correctly. Each one records what it saw on the way in. The CDN saw HTTPS, so it sets X-Forwarded-Proto: https. The origin server also received HTTPS — from the CDN — so it adds its own https.

Depending on the proxy, "adds" means either a second header line or a comma-appended value. From the application's point of view it makes no difference, and that is the first thing worth measuring.

Two header lines are one string

The Fetch Headers object, which is what a Next.js middleware receives, combines repeated header lines into a single comma-separated value:

const h = new Headers();
h.append("x-forwarded-proto", "https");
h.append("x-forwarded-proto", "https");

h.get("x-forwarded-proto");
// "https, https"

So there is no way to "just read the first one" through .get(). Whatever sits between you and the client, you get the list.

Where it breaks

The site used Auth.js v5 (next-auth beta) with its auth() wrapper around the middleware. The wrapper is convenient: it resolves the session for every request and hands it to your middleware as req.auth.

To resolve the session, it builds an internal URL for its own session endpoint. When no AUTH_URL environment variable is set, that URL is assembled from request headers. The relevant part of createActionURL in @auth/core reads:

const detectedHost = headers.get("x-forwarded-host") ?? headers.get("host");
const detectedProtocol = headers.get("x-forwarded-proto") ?? protocol ?? "https";
const _protocol = detectedProtocol.endsWith(":")
  ? detectedProtocol
  : detectedProtocol + ":";
url = new URL(`${_protocol}//${detectedHost}`);

With one proxy, that produces https://example.org. With two, it produces this string:

https, https://example.org

which is not a URL. Calling it directly, against the same library build:

Input	Result
single `https`	`https://example.org/api/auth/session`
`https, https` (two header lines)	`TypeError: Invalid URL`
`https, https` (one comma list)	`TypeError: Invalid URL`
`http, https`	`TypeError: Invalid URL`

The error is thrown inside the middleware, and the middleware runs on every matched path. So it is not "auth is broken" — it is the site that is broken. Every page, including ones that never look at the session, returns 500.

The development machine never showed it because it had no proxy in front at all, let alone two.

The obvious fix, and the second failure

Reading the code above, the escape hatch is clear: if AUTH_URL is set, the header branch is never taken. Measured:

Input	`AUTH_URL`	Result
`https, https`	unset	`TypeError: Invalid URL`
`https, https`	`https://example.org`	`https://example.org/api/auth/session`

So setting AUTH_URL removes the crash. But in the wrapper, AUTH_URL does a second thing. Before your middleware runs, the request itself is rebuilt with the configured origin:

export function reqWithEnvURL(req) {
  const url = process.env.AUTH_URL ?? process.env.NEXTAUTH_URL;
  if (!url) return req;
  const { origin: envOrigin } = new URL(url);
  const { href, origin } = req.nextUrl;
  return new NextRequest(href.replace(origin, envOrigin), req);
}

Inside the application, the request used to say http://127.0.0.1:3000/.... Now it says https://example.org/.... Any later middleware that builds a rewrite from req.url — the i18n locale rewrite, in this case — now rewrites to the public address instead of the local one. A rewrite to an external origin is a proxy hop back out through the CDN, into the same middleware, which rewrites again.

The observed result on the live site was a redirect loop on /. The crash was gone; the site was still down.

What actually fixed it

The middleware never needed the full session. It needed one question answered — is there a valid admin token on this request? — to guard a handful of routes.

getToken from next-auth/jwt answers exactly that, and its implementation touches only two headers: cookie and authorization. It never parses the protocol, never parses the host, never builds a URL:

import { getToken } from "next-auth/jwt";

const token = await getToken({
  req,
  secret: process.env.AUTH_SECRET,
  secureCookie: true,
});

Replacing the auth() wrapper with getToken in the middleware fixed both failures at once. AUTH_URL stays in the environment, because the route handlers under /api/auth/* still use it — it just no longer rewrites every request passing through middleware.

The `secureCookie` line is not optional

getToken has to know the cookie's name, and the name depends on whether the session was issued over HTTPS. Measured from the library's cookie defaults:

`secureCookie`	Cookie name it looks for
`false` (default)	`authjs.session-token`
`true`	`__Secure-authjs.session-token`

Behind TLS, the login flow issues the __Secure- cookie. Without secureCookie: true, getToken looks for the other name, finds nothing, and returns null — no error, just "not logged in", forever. The flip side: with it hard-coded to true, admin login on a plain-HTTP local dev server will not work. That is the right trade-off for production, but it needs to be written down somewhere a teammate will find it.

The neighbouring headers have the same problem

Once one forwarded header is known to carry a list, the others deserve the same suspicion.

`X-Forwarded-For`: the last value is the CDN

Common advice for reading the client IP is "take the last value of X-Forwarded-For, because the client can forge the earlier ones." That advice assumes exactly one trusted proxy. With a CDN in front, the chain looks like this:

x-forwarded-for: 203.0.113.7, 198.51.100.20
                 └ client     └ CDN edge node

The last value is the CDN's edge node. Every visitor routed through that node shares it. A per-IP rate limiter or login lockout keyed on it doesn't limit one abuser — it limits everyone at once, and one failed-password burst locks the whole region out of the login form.

This was caught in review before it shipped, but it is the same class of bug. The fix is to prefer the CDN's own client-IP header — Cloudflare sends CF-Connecting-IP, other CDNs have their own equivalent — and to fall back to X-Forwarded-For only when that header is absent. Only trust the CDN header if the origin actually refuses traffic that did not come through the CDN; otherwise anyone can set it.

Canonical redirects: read the first value, or the CDN's own signal

The same middleware issued an HTTP → HTTPS canonical redirect. It now reads the CDN's visitor-scheme header where available, and otherwise takes the first element of X-Forwarded-Proto — the hop nearest the browser — after splitting on the comma:

const proto = (req.headers.get("x-forwarded-proto") ?? "")
  .split(",")[0]
  .trim();

Reading the whole string and comparing it to "https" would have been false for every request, and redirected every request to itself.

Checklist for "works on staging, 500 behind the CDN"

Count your proxies. A setup with one proxy (or none) and production with two are different environments, even with identical code.
Dump the forwarded headers from production once — x-forwarded-proto, x-forwarded-host, x-forwarded-for — and look for commas.
Grep dependencies for code that turns those headers into a URL. Any new URL(\${proto}://${host}) is a crash waiting for a second proxy.
Keep middleware small. If it only needs "is there a token", read the token; don't resolve a whole session on every request.
Know what your auth environment variables do besides the thing you set them for. AUTH_URL fixed one failure and caused another.
Never key rate limits on the last X-Forwarded-For value behind a CDN.
Test through the full chain before cutover, not against the origin port directly.

The general shape

Nothing here is a bug in the usual sense. The CDN appended correctly. The origin server appended correctly. The Fetch API combined repeated headers exactly as specified. The auth library built a URL from the headers it was given. Each piece is correct on its own; the failure is in the assumption, three layers deep, that a header which is allowed to be a list will only ever hold one item.

Headers with Forwarded in the name describe a chain. Code that reads them as a single value works until the day the chain gets one link longer — and that day is usually the day you put the site behind a CDN, in production, in front of everyone.

Written at Alesta WEB, from a production cutover where every page returned 500 until one comma was accounted for.

Six Rows Deleted. Three Decrements Applied. No Error.

Mahmut Gündüzalp — Wed, 16 Sep 2026 06:52:54 +0000

A cached counter on a listing page was wrong. Not wrong in an interesting way — just quietly, slightly too high. The kind of number nobody audits because nobody imagines it can drift.

It had been drifting for months. Across thirteen installations of the same application, the reconciliation query eventually found roughly 83,000 rows where the stored counter disagreed with the actual count. No error log entry. No failed query. No exception. Every write had returned success, every day, for months.

The cause is a single line of SQL that reads correctly, executes correctly, reports success correctly, and does not do what it says.

The query

The cleanup job removes flagged replies and decrements the per-topic counter. One statement for the delete, one for the counter:

UPDATE topics t
  JOIN replies r ON r.topic_id = t.id
  SET t.reply_count = t.reply_count - 1
WHERE r.spam = 1;

DELETE r FROM replies r WHERE r.spam = 1;

Read it out loud and it sounds right. For every spam reply, subtract one from its topic's counter. Then delete the replies.

It is not right. The UPDATE subtracts one. Total. Per topic. No matter how many replies matched.

The measurement

Three topics, nine replies, six of them flagged:

CREATE TABLE topics (
  id INT PRIMARY KEY,
  reply_count INT NOT NULL DEFAULT 0
) ENGINE=InnoDB;

CREATE TABLE replies (
  id INT PRIMARY KEY AUTO_INCREMENT,
  topic_id INT NOT NULL,
  spam TINYINT NOT NULL DEFAULT 0,
  KEY (topic_id)
) ENGINE=InnoDB;

INSERT INTO topics VALUES (1,5),(2,3),(3,1);
INSERT INTO replies (topic_id,spam) VALUES
 (1,1),(1,1),(1,1),(1,0),(1,0),
 (2,1),(2,1),(2,0),
 (3,1);

Topic 1 has three flagged replies, topic 2 has two, topic 3 has one. Counters start at 5, 3, 1. After removing the flagged replies they should read 2, 1, 0.

Run the UPDATE:

id  reply_count
1   4
2   2
3   0

Topic 1 lost one instead of three. Topic 2 lost one instead of two. Topic 3 is correct — and that is the whole problem, because topic 3 is what a hand test looks like.

Identical numbers on two different MariaDB major versions. This is not a version quirk and not a bug. It is the documented behavior of a multi-table UPDATE: each target row is visited at most once per statement, regardless of how many joined rows satisfy the condition. The engine is not iterating the join result and applying your expression once per row. It is iterating the target table and asking "does any joined row match?"

The answer is yes or no. Not three.

Why nobody catches it in testing

This is the part worth internalizing, because the trap is not the semantics — it's the shape of the evidence.

A constant assignment behaves perfectly. Change the SET from an expression to a literal:

UPDATE topics t
  JOIN replies r ON r.topic_id = t.id
  SET t.reply_count = 0
WHERE r.spam = 1;

id  reply_count
1   0
2   0
3   0

Flawless. Every topic with at least one flagged reply gets zeroed. SET status = 'archived', SET flagged = 1, SET reviewed_at = NOW() — every one of these works exactly as written, because "visit the row once" and "set it to a constant" produce the same result whether you visit once or ten times.

Multi-table UPDATE ... JOIN is a pattern developers use dozens of times and it is correct in almost every one of them. It becomes wrong the moment the right-hand side of the assignment references the column being assigned. Nothing in the syntax marks that boundary.

The client reports identical success. I ran the broken query and the correct query through the same connection and read back the driver's counters:

wrong query -> affected_rows = 3 | Rows matched: 3  Changed: 3  Warnings: 0
right query -> affected_rows = 3 | Rows matched: 3  Changed: 3  Warnings: 0

Three topics matched, three topics changed, zero warnings. Both times. The broken statement and the correct statement are indistinguishable from the application's point of view. If your job logs "updated 3 topics", that log line is true and useless.

Now put the delete next to it:

DELETE -> affected_rows = 6

Six rows deleted, three decrements applied. That discrepancy is the only signal the database ever gives you, and it appears nowhere unless you deliberately compare the two numbers. Nobody compares them, because in a healthy job they are supposed to differ — you delete N replies and touch M topics, and M is naturally smaller. The number that is wrong looks exactly like the number that is right.

The drift compounds silently. Run the cleanup three times against a topic with three flagged replies:

run 1 -> reply_count = 4
run 2 -> reply_count = 3
run 3 -> reply_count = 2

One per run, forever, in the wrong direction from the truth. A nightly job doesn't produce one visible error; it produces a slow leak. By the time somebody notices the number looks odd, there is no incident to correlate it with — no deploy, no outage, no spike. Just a figure that has been wrong for a while.

The fixes that don't work

The instinct once you understand the cause is to make the UPDATE iterate the join instead of the target. You can't. I tried the three obvious routes:

JOIN + GROUP BY            REJECTED (syntax error)
UPDATE DISTINCT            REJECTED (syntax error)
JOIN + ORDER BY + LIMIT    ACCEPTED

The first two are refused outright. The third is accepted on MariaDB and is worth a warning: it parses, it runs, it returns success, and it does not change the "once per target row" rule at all. You get a statement that looks like it addresses the problem and doesn't. (MySQL rejects ORDER BY/LIMIT on a multi-table UPDATE — so this particular false trail is engine-dependent, which is worse than if it were universal.)

There is no flag. The aggregation has to happen before the UPDATE sees the rows.

The fix

Aggregate in a derived table, join against that, and subtract the count:

UPDATE topics t
  JOIN ( SELECT topic_id, COUNT(*) AS n
           FROM replies
          WHERE spam = 1
          GROUP BY topic_id ) g
    ON g.topic_id = t.id
  SET t.reply_count = t.reply_count - g.n;

id  reply_count
1   2
2   1
3   0

Now the target table is still visited once per row — that never changed — but the value it needs is already a single number. The join produces exactly one row per topic by construction.

If the counter is derivable at all, the sturdier option is to stop doing arithmetic on it and recompute it outright:

UPDATE topics t
  SET t.reply_count = ( SELECT COUNT(*)
                          FROM replies r
                         WHERE r.topic_id = t.id
                           AND r.spam = 0 );

id  reply_count
1   2
2   1
3   0

Same result, and with a property the incremental version can never have: it is idempotent. Run it twice, run it a hundred times, the answer is the answer. An incremental counter is a running total of every code path that ever touched it, including the ones that were wrong and the ones that crashed halfway. An absolute recount has no memory of any of that. It costs more, and on a table where the counter is read far more often than it is written, that cost buys you a number that cannot drift.

Use the derived-table form when a full recount is genuinely too expensive. Use the recount when it isn't. Do not use the plain - 1.

Finding the damage you already have

If this pattern is anywhere in your codebase, the counters are already wrong and have been for a while. The reconciliation query is cheap and worth running before you fix anything, so you know the size of the problem:

SELECT t.id,
       t.reply_count             AS stored,
       COUNT(r.id)               AS actual,
       t.reply_count-COUNT(r.id) AS drift
FROM topics t
LEFT JOIN replies r ON r.topic_id = t.id
GROUP BY t.id, t.reply_count
HAVING t.reply_count-COUNT(r.id) <> 0
ORDER BY ABS(t.reply_count-COUNT(r.id)) DESC;

id  stored  actual  drift
1   4       2       2
2   2       1       1

One detail that will bite you when you write this from memory: the expression in HAVING has to be repeated in full. Referencing the drift alias fails on MariaDB with Reference 'drift' not supported (reference to group function). Same for the ORDER BY. It is a thirty-second detour that reads like a typo when you hit it at the end of a long day.

LEFT JOIN matters too. An inner join silently drops every topic whose replies were all deleted — and those are exactly the rows most likely to be wrong.

The general shape

The defect here is not SQL trivia. It is a statement whose plain-English reading and whose actual semantics diverge, where every observable signal — return code, affected row count, warning count, error log — agrees with the reading and not with the semantics.

You cannot test your way out of that with assertions on success. The only test that catches it asserts on the value: delete three replies from one topic, then check the counter went down by three. That test takes four lines. It is the difference between finding this in an afternoon and finding it in 83,000 rows.

If a counter in your schema has no test that pins its arithmetic to a real count, assume it is wrong and go measure it. It is a five-minute query and the answer is rarely zero.

Written at Alesta WEB, from a real cleanup job that had been quietly subtracting one for months.

The Socket Was Open. The Site Was Dead for Fourteen Hours.

Mahmut Gündüzalp — Mon, 07 Sep 2026 06:59:17 +0000

A site had been down since 19:54 the previous evening. Nobody reported it. The monitoring did not fire. When I finally looked, the cause was one process that had outlived its parent and was holding a socket nobody could use.

Every individual check passed. DNS resolved. Port 443 accepted connections. The TLS handshake completed against a valid certificate with two months left on it. The web server was running and serving other sites on the same machine. And every request to that one site returned 503.

The failure lived in the gap between "the process exists" and "the process works," which is a gap most health checks do not cover.

What the checks reported

I started from the outside, because the outside is cheap to measure:

Layer	Check	Result
DNS	resolves to the expected address	fine
TCP	ports 80 and 443 accept	fine
TLS	handshake, chain, expiry	fine, 68 days left
HTTP (port 80)	`GET /`	302 to the HTTPS URL
HTTPS	`GET /`	first a 20-second hang, then 503
Web server	running, other sites on the box served fine	fine

Two of those rows are the reason nobody noticed.

The port-80 row returns 302 because the HTTP-to-HTTPS redirect is a server-level rewrite. It never reaches PHP. A monitor watching http:// sees a chain that starts with a perfectly healthy 302, and if it treats a redirect as up, it reports up. Forever. The application behind it can be completely dead.

The TLS row is worse, because it looks like the most meaningful check in the list. A successful handshake proves the listener is alive and the certificate is valid. It proves nothing whatsoever about whether anything can generate a response body.

The first HTTPS request I made hung for twenty seconds and timed out. The second returned 503 in 143 milliseconds. That inconsistency is itself a clue: the server was trying to hand the request to a backend, waiting, giving up, and eventually not bothering to wait at all.

The log line that mattered

The site's own error log was useless. Its last entry was seventeen days old, because the log level was WARN and this condition never produced a warning at the virtual-host level.

The server-wide log had it, at INFO:

[INFO] [...#site] Connection idle time too long: 301 while in state: 6 ...
[INFO] [...#site] HttpExtConnector state: 0, request body sent: 0,
                  response body size: -2, response body sent: 0, attempts: 0.
[INFO] [...#site] External processor is not available.
[NOTICE] [...#site] oops! 503 Service Unavailable

External processor is not available is the whole diagnosis in five words. The web server had a request, knew which backend should handle it, and could not get a usable connection to that backend. In LSAPI terms the external processor is the PHP process pool. In PHP-FPM terms it is the pool behind the socket you point fastcgi_pass at. Same shape, same failure mode.

Note attempts: 0 and response body size: -2. It did not fail partway through a response. It never got one started.

PPID 1

The pool for this site is configured for ten children and ten concurrent connections. I expected either zero processes — pool dead, should have respawned — or ten busy ones.

There was exactly one:

  PID   PPID STAT     ELAPSED     TIME %CPU   RSS CMD
793380      1 SN      13:52:58 00:07:31  0.9  7036 lsphp

Read that line carefully, because it contains the entire bug.

PPID 1. The parent is gone. This process was adopted by init. A pool child whose master has died is an orphan, and an orphan is supervised by nothing. Nothing will restart it, nothing will kill it, nothing will notice it.

ELAPSED 13:52:58 against TIME 00:07:31. Fourteen hours of wall clock, seven and a half minutes of CPU. It was not spinning. No load-average alert was ever going to catch this.

STAT SN — interruptible sleep, low priority. The kernel confirms it:

$ grep State /proc/793380/status
State:  S (sleeping)

$ cat /proc/793380/wchan
hrtimer_nanosleep

hrtimer_nanosleep means the process is inside a timed sleep. Some PHP path called sleep() and the value it was handed put the process to bed for what may as well be forever. An API retry backoff, a lock-wait loop, a scraper delay — the specific call matters less than the shape: a wait nobody bounded.

Its open file descriptors closed the case:

$ ls -l /proc/793380/fd
0 -> socket:[598208598]
1 -> /usr/local/.../stderr.log (deleted)
2 -> /usr/local/.../stderr.log (deleted)
3 -> socket:[598208598]

Two descriptors on the listening socket. And stdout and stderr pointing at a log file marked (deleted) — something removed that file while the process held it open, so every byte it wrote for fourteen hours went to an inode with no name. If it printed a fatal error, a stack trace, a reason, that is gone. Never let cleanup delete a log a live process still holds open; rotate it and signal the writer instead.

Why nothing restarted it

Here is the part that generalizes past any one server.

A process supervisor decides whether to start a pool by asking two questions: does the socket file exist, and is the PID in the pid file alive?

srwxr-xr-x 1 ... /tmp/.../siteuser.sock
-rw-r--r-- 1 ... /tmp/.../siteuser.sock.pid

Both answers were yes. The socket file was on disk with the right permissions. The pid file named a process that was alive — sleeping, low priority, adopted by init, but very much running.

So the supervisor concluded the pool was up and never spawned a replacement. Meanwhile the orphan, asleep inside a timer, accepted nothing. Every incoming connection landed on a socket with no reader, waited out the connect timeout, and became a 503.

The pool was neither alive nor dead. It was occupying the identity of a working pool — the socket path and the pid — without doing any of the work. A liveness check that asks "is the PID alive" cannot see this, and neither can anything watching for a crash, because nothing crashed.

Fourteen hours

I went back to the access log and counted requests per hour:

That is the moment of death, and it matches the process start time to within a few minutes. From 20:00 onward the only requests left are bots, retries, and the handful of people who tried, got nothing, and left.

Nineteen requests an hour is not zero, and that is the trap. A "traffic dropped to zero" alert never fires. A daily-total alert compares a bad day against a normal one and shrugs at a 90% drop if the threshold is 95%. The site served errors to every visitor for fourteen hours while the graph showed a line that was low but not flat.

The fix, and it is boring

kill -TERM 793380
rm -f /tmp/.../siteuser.sock /tmp/.../siteuser.sock.pid

Kill the orphan, remove the stale socket and pid file so the supervisor stops believing a pool exists, then send one request. The next request rebuilt the pool by itself. Local response: 200 in 0.171 seconds. From outside: 200 in 0.29 seconds, full page, no error signatures in the HTML. Four healthy children within seconds.

Under ten seconds of repair after fourteen hours of downtime. That ratio is the actual story. The failure was trivial to fix and completely invisible.

What to check instead

Monitor the URL users actually load, and follow redirects to the end. A 302 from a server-level rewrite is not evidence that your application runs. If your monitor reports on the first response in the chain, it is monitoring your rewrite rules.

Assert on the body, not the status code. Match a string that only appears when the application really rendered — a title, a footer, a token in the markup. Byte length works too: a 503 page is a few hundred bytes, a real page is tens of kilobytes.

Alert on relative traffic collapse, not on zero. A drop from 306 to 19 requests in one hour is a far better signal than any absolute floor, and it fires within the hour instead of the next morning.

Treat long-lived pool children as suspect. Pool workers are meant to be recycled. Any child running for hours with almost no CPU deserves a look. ps -eo pid,ppid,etime,time,cmd plus /proc/<pid>/wchan costs nothing in a cron job, and a worker with PPID 1 is an orphan by definition.

Cap every sleep and every outbound call. The root cause under all of this is an unbounded wait. Timeouts on HTTP clients, bounded retry backoffs, lock waits that eventually give up. A worker allowed to sleep forever will eventually do exactly that.

Never delete a log file a process still holds open. Rotate and signal. Otherwise the one artifact that would have told you why is writing into an unlinked inode.

The short version

DNS, TCP, TLS and the certificate can all be perfect while the site returns 503 to every visitor.
External processor is not available means the web server could not get a working connection to the PHP pool. It is a backend problem, not a network or certificate problem.
A pool child whose parent has died gets PPID 1 and is supervised by nothing.
If that orphan keeps the socket and pid file, the supervisor sees a live pool and refuses to spawn a real one. Neither alive nor dead, just occupying the identity.
/proc/<pid>/wchan naming a sleep function, with hours of elapsed time and seconds of CPU, is the confirmation.
Health checks that follow redirects and assert on body content would have caught this in the first minute.

The site was down for fourteen hours. The fix was one signal and two deleted files. Everything expensive about this outage happened in the gap between a process existing and a process working.

Written from production work at Alesta WEB.

Lowercasing "İ" Adds a Character: How Turkish Text Broke a Keyword Router

Mahmut Gündüzalp — Fri, 04 Sep 2026 06:41:21 +0000

A classifier was sending questions to the wrong bucket. Not occasionally. I measured it against twenty real inputs and seven of them landed somewhere they did not belong. Thirty-five percent.

The routing code was thirty lines long and did the obvious thing: lowercase the incoming text, then check whether any keyword for a category appears in it. First match wins.

Every one of those thirty lines was wrong in a way that only shows up in Turkish, and two of the three problems will bite you in any language whose case mappings are not one-to-one.

The symptom

Users type free-form questions. The router picks a topic so that the right prompt and the right reference data get loaded. When it picks wrong, the answer is confidently about the wrong subject, which is worse than no answer.

The reported complaint was "the answers feel shallow." The instinct is to blame the model. The model was fine. It was being handed the wrong context.

Trap 1: lowercasing a dotted capital I produces two code points

Here is the whole thing, measured rather than remembered:

$s = "İş";                              // "work", capital İ
$l = mb_strtolower($s, "UTF-8");

// original      : U+0130 U+015F
// mb_strtolower : U+0069 U+0307 U+015F
// mb_strlen     : 2 -> 3

U+0130 LATIN CAPITAL LETTER I WITH DOT ABOVE has no single-code-point lowercase form in the default Unicode case mapping. It maps to U+0069 LATIN SMALL LETTER I followed by U+0307 COMBINING DOT ABOVE. The string got longer. It still renders as "i̇ş" and looks entirely normal in your terminal, your editor, and your database client.

So this happens:

mb_strpos(mb_strtolower("İşten çıkarılır mıyım", "UTF-8"), "iş");   // false
mb_strtolower("İSTANBUL", "UTF-8") === "istanbul";                  // false

Your keyword is two code points. The haystack has three. No match, no error, no warning. The comparison returns false and the request goes to whatever category happens to be checked next.

The reason this is not an edge case: people capitalize the first word of a sentence. In Turkish a large number of everyday words begin with İ — the words for work, request, relationship, reputation, tender, transaction, need, cancellation. Any of them starting a sentence produces the three-code-point form. The broken form is the common form, and the version your test uses — all lowercase, typed by you — is the rare one.

That is why this survives a green test suite. You wrote "işten çıkarılır mıyım" in the fixture, in lowercase, by hand. It matches. The user wrote "İşten çıkarılır mıyım". It does not.

Unicode normalization will not save you

The reflex is to reach for Normalizer::normalize() and move on. I checked:

lower    : U+0069 U+0307 U+015F
NFC      : U+0069 U+0307 U+015F
NFKC     : U+0069 U+0307 U+015F
casefold : U+0069 U+0307 U+015F

NFC composes a base character with a combining mark when a precomposed code point exists. There is no precomposed "latin small letter i with dot above" — a lowercase i already has a dot, so Unicode never needed one. The sequence stays two code points under every composing form, and casefold() does not touch it either.

NFD is more interesting, because it goes the other way and decomposes everything:

NFD("i̇ş") : U+0069 U+0307 U+0073 U+0327

Now the cedilla under ş is its own code point too. That gives you one rule that handles both problems at once: decompose, then drop every nonspacing mark.

import unicodedata as u

def norm(s: str) -> str:
    s = u.normalize('NFD', s.lower())
    s = ''.join(c for c in s if u.category(c) != 'Mn')
    return s.replace('ı', 'i')      # dotless ı has no combining mark to strip

Measured output:

'İş'         -> 'is'
'İSTANBUL'   -> 'istanbul'
'Değişiklik' -> 'degisiklik'
'İlişki'     -> 'iliski'
'Ölüm'       -> 'olum'

In PHP, where the intl extension is frequently not loaded, the same thing without a normalizer:

function tr_norm(string $s): string {
    $s = mb_strtolower($s, 'UTF-8');
    $s = str_replace("\u{0307}", '', $s);        // the leftover dot from İ
    return strtr($s, [
        'ı'=>'i','ş'=>'s','ğ'=>'g','ü'=>'u','ö'=>'o','ç'=>'c',
        'â'=>'a','î'=>'i','û'=>'u',
    ]);
}

The single most important line is the U+0307 removal. Delete it and the bug is back.

Run your keyword list through the same function, too. Both sides of a comparison must see the same normalization, always. A keyword table normalized at author time and input normalized at request time will drift the moment somebody edits the table by pasting from a document.

The other direction, for readers who do not write Turkish

Turkish has two distinct letters where English has one: dotted İ/i and dotless I/ı. They are different letters, not styling. So the correct Turkish lowercase of I is ı, not i.

This is the well-known "Turkish I problem," and it usually appears as the mirror image of what I described. In Java, .NET, or any locale-sensitive toLowerCase() running under a tr-TR locale:

"INFO".toLowerCase()          // "ınfo" under tr-TR, "info" everywhere else
"FILE".toLowerCase()          // "fıle"

Code that lowercases a protocol keyword, a header name, a file extension, or an SQL identifier and then compares it against an ASCII literal breaks on a machine whose locale happens to be Turkish. The fix there is the opposite of the one above: pass an explicit invariant locale, toLowerCase(Locale.ROOT), and never let ambient locale decide the semantics of a comparison.

So there are two rules, pointing in opposite directions depending on what you are doing:

Comparing protocol tokens and identifiers: force a locale-invariant mapping. Never fold, never guess.
Comparing human text for search or classification: normalize aggressively, on both sides, with a function you control.

Confusing the two is how you end up with a system that is broken in one country and subtly wrong everywhere else.

Trap 2: unbounded substring matching

With normalization fixed, the router still misrouted. The second bug had nothing to do with Unicode: the check was strpos($text, $keyword) !== false, with no word boundary.

Short keywords swallow longer words that merely contain them. Real collisions from the keyword table:

Keyword	Means	Also matches	Which means
`olum`	death	olumlu, olumsuz	positive, negative
`mal`	goods, property	normal, mal**zeme	normal, material
`is`	work	kisi, degisiklik, gise	person, change, counter
`ask`	love	maske	mask
`din`	religion	aydin, din**len	bright, rest

olum was the expensive one. "Olumlu" — "positive" — is among the most common words in any yes/no question. Every such question was routed to the death-and-inheritance category, because that category happened to sit near the top of the map.

Word boundaries in an agglutinative language are not just \b around a fixed string. Turkish stacks suffixes, and stems mutate: "lost" appears as kayıp, kaybettiğim, kaybolan — the stem itself shifts from kayıp to kayb. So match a stem plus a bounded run of following letters:

$pattern = '/(?<!\p{L})' . preg_quote($stem, '/') . '\p{L}{0,6}(?!\p{L})/u';

(?<!\p{L}) and (?!\p{L}) are Unicode-aware boundary assertions — plain \b reasons about \w, whose meaning depends on flags and locale. The {0,6} allows suffixes without letting the stem float in the middle of an unrelated word. And keep a stem list per concept rather than one string: kayıp|kayb|kaybol|kaybet, because sound changes will not occur to you at the keyboard, only in production.

Trap 3: map order is a priority ranking you did not know you wrote

foreach ($map as $topic => $words) {
    foreach ($words as $w) {
        if (matches($text, $w)) return $topic;   // first hit wins
    }
}

Whatever order the array literal happens to be in is your precedence rule. "Will my father's health improve" contains both a family keyword and a health keyword. Family was declared fourth, health sixth, so it went to the family bucket every time.

Two changes fix the class of problem rather than the instance:

Order specific before general, deliberately, with a comment saying so.
Stop returning on first hit. Collect every match, score them, take the best. A reasonable score is the number of distinct stems matched, weighted by stem specificity — a rare six-letter stem is worth more than a three-letter one that appears everywhere.

Scoring also gives you something first-match never can: a confidence number. When the top two topics score within noise of each other, you can fall back to a neutral path instead of committing to a coin flip.

Normalize for matching, never for identity

One caution, because the recipe above is lossy on purpose:

'ILIK'  -> 'ilik'
'ılık'  -> 'ilik'

Two different words collapse to the same key. That is exactly what you want for search and classification, and exactly what you must never do for identity.

If you normalize usernames or email local parts this way before a uniqueness check or a login comparison, you have created an account-collision path: two visually and semantically distinct identifiers now compare equal. The same applies to case-insensitive uniqueness in general — the database collation, the application-level check and the login comparison have to agree on one rule, and that rule should be locale-invariant, not a fuzzy fold.

Fold for relevance. Compare exactly for identity. Keep the two functions in separate files so nobody reuses one for the other by accident.

How to verify, because you cannot eyeball this

Every symptom in this article is invisible on screen. The three-code-point string looks identical to the two-code-point one. You cannot review your way to correctness here; you have to measure.

What I did, and would do again:

Write a table of 20–30 real inputs with the expected category, taken from actual traffic rather than invented. Run it, print the wrong-rate as a number. "Seven of twenty" is a fact you can act on; "seems better now" is not.
Sweep for collisions mechanically. For every keyword shorter than about five characters, generate common words containing it and feed them through the classifier. That is how olum inside olumlu surfaced — nobody was going to catch it by reading.
Print code points, not strings, whenever a comparison fails for no visible reason. printf("U+%04X ") over the characters takes ten seconds and ends the argument.
Re-run the same table after the fix and report the new number. A fix that is not re-measured is a hypothesis.

The short version

mb_strtolower('İ') returns i + U+0307. The string gets longer, looks the same, and stops matching.
NFC and casefold do not collapse it. NFD plus stripping nonspacing marks does.
Normalize the input and the keyword list with the same function.
Match stems with Unicode-aware boundaries, not bare substrings.
Map order is silent precedence. Score instead of returning on first hit.
Fold for search; compare exactly for identity.
Measure the wrong-rate before and after, on real inputs.

None of this is exotic Unicode trivia. It is one line of case mapping and one missing word boundary, and together they were quietly discarding a third of the routing decisions in a system that looked, from the outside, like a model quality problem.

Written from production work at Alesta WEB.

The Word "null" Is Not a Null: How JSON Nulls Turn Into Text on Android

Mahmut Gündüzalp — Sun, 30 Aug 2026 10:39:10 +0000

A user opened our mobile app and saw this on their profile screen:

Membership: null
Points: null
Last payment: null

The same account on the website showed empty fields and a zero. No crash, no error dialog, no red line in the log. The app had received the data, parsed it successfully, and printed the four characters n, u, l, l into a TextView.

The parsing code looked like it had already handled this:

val plan = obj.optString("membership", "")

That is the whole bug. On Android, when membership is JSON null, optString does not return "". It returns the string "null".

Why it does that

org.json on Android represents JSON null with a sentinel object, JSONObject.NULL. It is not Java null — it is a real object that exists specifically so the library can tell "the key is absent" apart from "the key is present and its value is null."

Android's implementation of optString is, in essence:

public String optString(String name, String fallback) {
    Object object = opt(name);
    String result = JSON.toString(object);   // ~ object.toString()
    return result != null ? result : fallback;
}

opt("membership") returns the sentinel. The sentinel is not Java null, so the fallback branch is never reached. toString() is called on it, and JSONObject.NULL.toString() returns — by design, because it has to serialize back to valid JSON — the text "null".

The fallback works exactly as documented for a missing key. It does nothing for a null key. Those are two different states and the API gives them one parameter.

getString has the same outcome. It does not throw on a null value; it hands you "null" as well.

Now compare the neighbours:

obj.optInt("points", 0)        // JSON null -> 0     ✅
obj.optBoolean("active", false) // JSON null -> false ✅
obj.optString("plan", "")       // JSON null -> "null" ❌

optInt and optBoolean route through converters that return Java null for the sentinel, so the fallback fires. Only the string path stringifies the sentinel. This inconsistency is why the bug survives review: three lines that look identical, and one of them behaves differently.

It also depends on which `org.json` you are compiled against

This is worth checking rather than assuming. The reference implementation from json.org — the org.json:json jar you would pull into a plain JVM or server-side project — has a version of the same method that reads roughly:

public String optString(String key, String defaultValue) {
    Object object = this.opt(key);
    return NULL.equals(object) ? defaultValue : object.toString();
}

It checks the sentinel explicitly and returns your default. Same package, same class, same method signature, different answer.

Android ships its own Harmony-derived implementation in the framework, and it wins on the classpath. So a unit test that runs on the JVM with the json.org jar can pass while the device fails. If your test source set is green and your androidTest source set is red on the same assertion, this is a candidate.

Confirm it in ten seconds with an instrumented test rather than trusting either the docs or this article:

@Test fun optStringOnJsonNull() {
    val o = JSONObject("""{"a": null}""")
    assertEquals("", o.optString("a", ""))   // fails on device
}

Where the null came from in the first place

Fixing the client is only half of it, because the response should not have contained nulls for those fields at all.

The API was PHP, and the payload was built from a query that looked something like:

SELECT
    m.plan_name,
    SUM(p.amount)  AS total_paid,
    MAX(p.paid_at) AS last_payment
FROM members m
LEFT JOIN payments p ON p.member_id = m.id
WHERE m.id = ?
GROUP BY m.id

Two separate null factories in one query:

LEFT JOIN with no matching rows. Every column from payments comes back NULL.
Aggregates over an empty set. SUM() and MAX() return NULL, not 0 and not '', when there are no rows to aggregate. COUNT() is the exception — it returns 0 — which is exactly why people assume the others do too.

The website never showed this because the view layer cast on the way out:

echo (int)($row['total_paid'] ?? 0);
echo htmlspecialchars($row['plan_name'] ?? '');

The ?? 0 there is doing real work — ?? catches SQL NULL once PDO has turned it into PHP null. Web output was clean. The JSON endpoint serialized the same row directly, json_encode mapped PHP null to JSON null, and the app printed the word.

This is the general shape: a null that is invisible in one rendering path becomes visible in another. Two consumers of the same data, one of which happens to have a defensive cast in its template.

Why the test accounts never caught it

The seeded test users all had payment history. Every demo account had rows. The only way to produce this response was to log in as a member with nothing — a brand new registration, or an account whose records had been archived.

A user with zero rows is a boundary condition, and it is the one that seed scripts systematically do not create, because seed scripts exist to make screens look populated. If you write fixtures, make one account deliberately empty and put it in the smoke test.

Fix both layers, not one

The tempting move is to fix the server, since that is where the null originates, and declare the client fine. That is wrong for mobile specifically, and here is why.

Old app versions live forever. You control your server. You do not control which build is on a user's phone. A client that renders "null" when handed a JSON null is a permanently loaded gun pointed at every future endpoint, including ones written by someone who has never heard of this bug.

Caches outlive deploys. An HTTP cache, a CDN, or a local response cache can hand the app a pre-fix payload well after the server is patched.

New fields arrive constantly. Server-side normalization is a policy you have to remember to apply. Client-side hardening is a helper you cannot forget to use, because it is the only way to read a string.

So: both.

Server side — normalize once, at the boundary

Recursively convert nulls at the point where the response is serialized, not in each handler:

function json_clean(mixed $value): mixed
{
    if (is_array($value)) {
        foreach ($value as $k => $v) {
            $value[$k] = json_clean($v);
        }
        return $value;
    }
    return $value ?? '';
}

// single exit point for every API response
function api_respond(array $payload, int $status = 200): void
{
    http_response_code($status);
    header('Content-Type: application/json; charset=utf-8');
    echo json_encode(json_clean($payload), JSON_UNESCAPED_UNICODE);
}

One caveat worth stating plainly: this flattens the difference between "no value" and "empty value". For a display API that is what you want. For an API where a client needs to distinguish "field not provided" from "field explicitly cleared" — a PATCH-style contract, for instance — do not do this globally. Fix those fields at the query instead:

COALESCE(SUM(p.amount), 0)   AS total_paid,
COALESCE(m.plan_name, '')    AS plan_name

COALESCE is the honest fix; the recursive normalizer is the safety net under it.

Client side — never call `optString` directly again

Wrap it once, ban the raw call in review:

object Json {

    fun text(obj: JSONObject?, key: String, fallback: String = ""): String {
        if (obj == null || !obj.has(key) || obj.isNull(key)) return fallback
        val value = obj.optString(key, fallback).trim()
        // a server can also send the *string* "null" — from string concatenation,
        // a logging wrapper, or an older client that echoed this bug back
        if (value.isEmpty() || value == "null" || value == "undefined") return fallback
        return value
    }

    fun number(obj: JSONObject?, key: String, fallback: Long = 0): Long {
        if (obj == null || obj.isNull(key)) return fallback
        return obj.optLong(key, fallback)
    }

    fun textOrNull(obj: JSONObject?, key: String): String? =
        text(obj, key, "").ifEmpty { null }
}

isNull(key) is the load-bearing line. It returns true for both Java null and the JSONObject.NULL sentinel, which is the check optString fails to make on your behalf.

Filtering the literal "null" string looks paranoid until you have seen a server do "Welcome, " + name in a template and store the result. Once a "null" has been written to a database by an upstream system, no amount of correctness downstream removes it — you have to strip it at read time.

Then make the raw call unusable:

// build.gradle.kts — fail the build, not the user
// (or use Detekt's ForbiddenMethodCall rule)

A lint rule beats a code review comment, because the review comment only catches the calls written while the reviewer remembers the bug.

What about Gson, Moshi, kotlinx.serialization?

Mostly they handle this correctly, with one trap of their own.

Moshi and kotlinx.serialization are aware of Kotlin nullability. Declare val plan: String as non-null and feed it JSON null, and they throw at parse time — a loud, immediate, correctly-located failure. That is the behaviour you want.

Gson does not. It constructs objects through unsafe reflection and bypasses Kotlin's null checks entirely, so a JSON null lands in a field the type system swears is non-null. Nothing fails at parse time. The NPE happens later, somewhere unrelated, and the stack trace points at the innocent code that touched the field rather than the parser that lied about it. That is arguably worse than "null" on screen, because at least "null" on screen tells you where it came from.

If you are on Gson, declare the fields nullable (String?) and handle the nulls explicitly. Fighting for non-null fields you cannot enforce buys nothing.

org.json remains everywhere despite all three of these, for a simple reason: it is in the Android framework, weighs nothing, and needs no annotation processor. Small apps and old modules keep using it. It works fine — as long as you know that its string path does not have a null story.

The five-minute audit

Worth running on any app that talks to an API you also control:

grep -rn "optString(" app/src — every hit is a candidate. Replace with a helper that calls isNull first.
grep -rn "getString(" app/src on JSONObject receivers — same problem, no fallback parameter to give false comfort.
In the API, find aggregates without COALESCE: SUM(, MAX(, MIN(, AVG( — and every LEFT JOIN selecting columns from the right-hand table.
Create one test account with zero related rows and open every screen with it. This finds more null bugs in ten minutes than reading code for a day.
grep your database for the literal string 'null' in text columns. If you find any, something upstream has already been concatenating nulls into text, and that data needs cleaning independently of any code fix.

The underlying lesson is not about one library method. It is that null crosses system boundaries badly. SQL NULL, PHP null, JSON null, Java null, Kotlin null?, and a sentinel object that stringifies to "null" are six different things wearing the same name, and every hop between two of them is a place where one quietly becomes another. Put the conversion in one function per boundary, and you only have to be right once.

I build and run news, e-commerce and membership platforms at alestaweb.com. This one took an afternoon and a screenshot from a user who assumed "null" was our membership tier.

MySQL Will Quietly Eat Your Data If You Let It: Truncation, Charsets, and the 1366 You Never See

Mahmut Gündüzalp — Sun, 23 Aug 2026 08:30:21 +0000

A WhatsApp button on a shop was dialling the wrong number. Not a formatting problem — a genuinely different number, three digits short of the one in the admin panel.

The admin panel had saved it. The page was reading it. Nothing errored. The column was varchar(10), the number arrived with a country code, and MySQL had done exactly what it was configured to do: chop the value to fit and carry on.

That column had been wrong for months. Nobody noticed because nothing anywhere in the stack raised its voice.

Here is the family of bugs that behaves this way, how to find them in a database you inherited, and why the obvious fix — "just turn on strict mode" — creates a different bug if you do only that.

Part 1: The truncation

Run this on any MySQL or MariaDB instance:

SELECT @@GLOBAL.sql_mode, @@SESSION.sql_mode;

If neither contains STRICT_TRANS_TABLES or STRICT_ALL_TABLES, then for every INSERT and UPDATE your server is in the business of making data fit, not of telling you it doesn't.

What that means concretely:

CREATE TABLE t (phone VARCHAR(10));
INSERT INTO t VALUES ('905551234567');
SHOW WARNINGS;
-- Level: Warning  Code: 1265  Message: Data truncated for column 'phone' at row 1

SELECT phone FROM t;
-- 9055512345

Note the level. Warning, not error. The statement succeeded. affected_rows is 1. PDO in exception mode throws nothing, because there is nothing to throw — MySQL considers this a completed statement. Your ORM reports success. Your integration test asserting "the row exists" passes. The value is wrong.

The same mechanism applies well beyond strings:

You wrote	Column	Non-strict result
`'905551234567'`	`VARCHAR(10)`	`'9055512345'`
`300`	`TINYINT`	`127`
`'2026-02-31'`	`DATE`	`'0000-00-00'`
`''`	`INT NOT NULL`	`0`
`12.999`	`DECIMAL(4,2)`	`13.00`

Every one of those is a silent, permanent difference between what your application believed and what your database holds. And unlike a crash, there is no timestamp to correlate against — you cannot tell from the row when it happened or how many rows before it went the same way.

Finding the damage in a database you inherited

You cannot recover truncated values — the tail is gone. But you can find the columns where it is happening, which is what matters going forward. Truncated values pile up at exactly the column limit, so the suspects are the columns where real rows sit at the maximum length:

-- 1. list the string columns and their limits
SELECT TABLE_NAME, COLUMN_NAME, CHARACTER_MAXIMUM_LENGTH AS max_len
FROM information_schema.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
  AND DATA_TYPE IN ('varchar', 'char')
  AND CHARACTER_MAXIMUM_LENGTH <= 64      -- short columns are where this bites
ORDER BY CHARACTER_MAXIMUM_LENGTH;

Then, per interesting column:

SELECT COUNT(*) AS at_the_limit
FROM settings
WHERE CHAR_LENGTH(phone) = 10;

A handful of rows sitting exactly at the limit can be a coincidence. A majority of rows sitting exactly at the limit is not a coincidence, it is a report. In our case the count was every row that had ever been saved with a country code.

Two details that matter when you run this:

Use CHAR_LENGTH(), not LENGTH(). LENGTH() counts bytes; CHAR_LENGTH() counts characters. On multi-byte text they disagree, and VARCHAR(n) limits characters, not bytes.
Do this on a replica or a restored dump if the table is large. CHAR_LENGTH() in a WHERE clause means a full scan.

Part 2: Strict mode is necessary and not sufficient

The obvious response is to turn strict mode on. Do it — but understand what you are buying.

Strict mode does not repair the mismatch between what your application sends and what the column accepts. It changes how you find out about it: instead of quietly storing a wrong value, MySQL raises error 1406 (Data too long for column) and your code, which never expected a write to fail, hands the user a 500.

I have seen exactly this on an admin form. Strict mode was correctly enabled; the title field in the form had no maxlength and no server-side length check; the column was varchar(150). Someone pasted a long headline. Instead of a validation message the editor got a blank error page, and — worse — a story they thought they had saved and hadn't.

That is still better than silent corruption, because it is loud and it is immediate. But the actual fix is at the boundary:

$rules = [
    'title' => ['max' => 150],   // same number as the column, in one place
    'phone' => ['max' => 20],
];

foreach ($rules as $field => $rule) {
    if (mb_strlen($input[$field] ?? '') > $rule['max']) {
        return back()->withError("{$field}: at most {$rule['max']} characters.");
    }
}

Three habits that keep this honest:

Validate before you write, with the column's real limit. Generate the limits from information_schema if you can — a hand-copied number drifts the first time someone runs an ALTER.
Turn strict mode on so the boundary check has a backstop. Validation you wrote can be bypassed; the database is the last honest party in the chain.
Widen the column when the data is legitimately bigger. VARCHAR(10) for a phone number was never right. Truncation is a symptom; a wrong schema is the disease.

Part 3: Charsets, where "it worked on the old server" lives

A database was copied to a new server. Everything imported. Row counts matched. A week later somebody noticed that Turkish characters in older records had turned into question marks in some tables and not others.

The mistake was trusting the database default:

SHOW CREATE DATABASE app;
-- CHARACTER SET utf8mb4  ← looks fine, means almost nothing

The database default applies to newly created tables that do not specify their own. It says nothing about the columns you already have. The columns are where the truth is:

SELECT TABLE_NAME, COLUMN_NAME, CHARACTER_SET_NAME, COLLATION_NAME
FROM information_schema.COLUMNS
WHERE TABLE_SCHEMA = 'app'
  AND CHARACTER_SET_NAME IS NOT NULL
  AND CHARACTER_SET_NAME <> 'utf8mb4';

If that returns rows, you have a mixed-charset database, and every dump/restore across it is a chance to lose bytes.

Two more charset facts worth having in your head:

utf8 in MySQL is not UTF-8. The alias utf8 (historically utf8mb3) stores at most three bytes per character, so it cannot hold anything outside the Basic Multilingual Plane — emoji, some CJK extensions, some historic scripts. Insert one into a utf8 column and, in non-strict mode, MySQL truncates the value at the emoji and keeps whatever came before. utf8mb4 is real UTF-8. Use it everywhere, including the connection charset.

The connection charset is part of the pipeline. Your column can be utf8mb4, your data can be perfect, and your terminal or client can still show you mojibake because the session negotiated something else. Which leads to the most useful debugging habit in this whole article:

When you think data is corrupted, check the bytes

SELECT title, HEX(title), LENGTH(title), CHAR_LENGTH(title)
FROM articles WHERE id = 42;

HEX() is the ground truth. It is not affected by your client, your terminal font, or the connection charset. I have twice now "found" data loss that turned out to be a display artefact of the command-line client, and both times HEX() settled it in one query: the bytes were intact, the rendering was not.

The tell for genuine UTF-8 content is LENGTH() > CHAR_LENGTH(). If they are equal on text you know contains non-ASCII characters, something upstream already flattened it.

Part 4: The application-side version of the same bug

Databases are not the only layer that cuts strings. This is a summary excerpt in PHP:

$summary = substr($body, 0, 200);   // wrong

substr() counts bytes. On UTF-8 text, byte 200 lands in the middle of a multi-byte character about half the time, and you send MySQL a string ending in half a character. Then either:

the column is utf8mb4 and strict mode is on → error 1366, Incorrect string value, and you get a 500 from what looked like a formatting line; or
strict mode is off → MySQL drops the invalid tail silently, and your summaries are quietly one character shorter than you think, sometimes ending in a replacement glyph.

The fix is one letter and three characters:

$summary = mb_substr($body, 0, 200);   // counts characters

Whenever you see strlen, substr, strtoupper, or str_pad applied to user text, treat it as a bug report waiting to happen. Error 1366 in a log almost always traces back to one of them.

The checklist

Run these on any project you did not set up yourself. They take about ten minutes together.

SELECT @@GLOBAL.sql_mode; — is STRICT_TRANS_TABLES present? If not, plan to enable it after auditing lengths, not before.
List varchar columns with small limits, then count rows sitting exactly at the limit. That count is your truncation report.
List columns whose CHARACTER_SET_NAME is not utf8mb4. Fix them before the next migration, not during it.
Confirm the connection charset your application actually negotiates — the DSN, not the config file you hope it reads.
Grep the codebase for byte-based string functions applied to user content, and replace them with the mb_ versions.
When you suspect corruption, run HEX() before you conclude anything. Half the time the data is fine and the client is lying.

The theme across all of this: the database is willing to accept an approximation of what you gave it, and by default it will not argue. Make it argue. Then make sure your code is ready to hear it.

I build and run news and e-commerce platforms at alestaweb.com. Every example above cost somebody a real afternoon.

Your Uptime Monitor Says 200. Your Contact Form Has Been Dead for Three Days.

Mahmut Gündüzalp — Sun, 23 Aug 2026 08:27:02 +0000

The contact form on a company site I look after stopped producing leads. Not "fewer leads" — zero, for three days. No alert fired. The uptime monitor was green the entire time. The status page had a nice unbroken bar.

The monitor was checking the homepage. The homepage was fine. /contact had been returning a fatal error since a Tuesday afternoon deploy.

This is the most boring class of outage there is, and it is the one that costs the most, because nothing tells you it is happening. Below is what actually broke, the four ways a site can be broken while still answering 200, and the small set of checks I run now instead of pinging the root URL.

What broke

The site loads a list of helper files at boot. During an unrelated edit, that list was rewritten, and one entry did not survive the rewrite — a helper that supplied data to a few templates.

Most pages never touch that helper, so they kept rendering. One page called it in a template. That page fataled. In production, with display errors off, a fatal means an empty response body or a generic error page — depending on how the stack is configured, sometimes with a 500 status, sometimes with the web server's own branded page and a status you did not choose.

The error log had the answer on line one, at the exact timestamp of the deploy. Nobody looked, because nothing said to look.

Three days of a form that a campaign was actively driving traffic to. The traffic arrived. The page did not.

Four ways to be broken and still return 200

If your health check is curl -o /dev/null -w "%{http_code}" https://example.com/ you are testing one route, one time, for one property. Here are the failure modes that sail straight through it.

1. The route you don't check. A site is not a URL, it is a few dozen to a few hundred routes. Homepages are the least likely page to break, because everything touches them and everyone looks at them. The pages that break are the ones with one unusual dependency: the form page, the search page, the report that joins four tables.

2. The page renders, but a part of it silently didn't. Templates are forgiving by design. A partial that throws inside a try/catch, a widget whose data source returned an empty array, a block that renders only when a config key exists — none of these change the status code. The page looks right at a glance and is missing the thing that made it worth serving.

I found one of these in the same audit: a function that emitted structured data (Article, BreadcrumbList) existed, was correct, and was called from nowhere. Thirty news URLs had been building JSON-LD that never reached the HTML. Status: 200. Value delivered: none.

3. The page is fine and the form is not. This one is hard to see. A full-page HTML cache was turned on. HTML caches are dumb by nature: they store bytes. Those bytes included a CSRF token. Every visitor got the first visitor's token, every POST failed validation, and the failure looked to the user like a page reload with no message. Every page on the site: 200. Every form on the site: broken.

4. The error page returns 200. Custom error handlers that render a friendly "something went wrong" view and forget to set the status. Frameworks that catch late and fall back to a template. Reverse proxies serving a static maintenance page with the default status. Your monitor sees 200 and a body of some length, and reports health.

What to assert instead

The fix is not a bigger monitoring product. It is asserting the things you actually care about. Four checks, in increasing order of value.

Check every route, not one route

You already have the route list — it is in your router, your sitemap, or both. Walk it.

#!/usr/bin/env bash
# routes.txt: one path per line, from the sitemap or the router
FAIL=0
while read -r path; do
  code=$(curl -s -o /tmp/body -w '%{http_code}' "https://example.com${path}")
  size=$(wc -c < /tmp/body)
  if [ "$code" != "200" ] || [ "$size" -lt 2000 ]; then
    echo "FAIL ${path} status=${code} bytes=${size}"
    FAIL=1
  fi
done < routes.txt
exit $FAIL

The size floor is doing real work here. A fatal that returns 200 with an empty or near-empty body is caught by bytes, not by status. Pick the floor from your own smallest legitimate page, then subtract a little.

Assert content, not just bytes

One string per route, chosen to be the thing that page exists to do. The form page must contain a <form and a submit button. The article page must contain an <h1. The page with structured data must contain application/ld+json.

grep -q '<form' /tmp/body || echo "FAIL ${path}: no form in body"
grep -qi 'fatal error\|<b>Warning</b>\|stack trace\|Undefined variable' /tmp/body \
  && echo "FAIL ${path}: error signature in HTML"

That second grep catches the inverse case: the page that renders and prints a warning into the output where a visitor — or a crawler — can read it.

Diff the error log around every deploy

This is the cheapest high-value check in the list and almost nobody does it. Record the log size before the deploy, read the tail after.

BEFORE=$(stat -c%s /path/to/error.log)
# ... deploy ...
sleep 30
AFTER=$(stat -c%s /path/to/error.log)
if [ "$AFTER" -gt "$BEFORE" ]; then
  tail -c $((AFTER - BEFORE)) /path/to/error.log
  echo "New errors appeared during deploy — investigate before walking away."
fi

In our incident this would have printed the missing helper, by name, thirty seconds after the deploy that caused it. The information was there the whole time. Nothing surfaced it.

Round-trip the form

The only way to know a form works is to submit it. A synthetic submission is about twenty lines and can run hourly:

<?php
// 1. Fetch the page like a browser would, keeping cookies.
$ch = curl_init('https://example.com/contact');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_COOKIEJAR      => '/tmp/probe-cookies',
    CURLOPT_COOKIEFILE     => '/tmp/probe-cookies',
]);
$html = curl_exec($ch);

// 2. Pull the CSRF token out of the markup, exactly like a browser.
preg_match('/name="_token" value="([^"]+)"/', $html, $m);
$token = $m[1] ?? null;
if (!$token) { exit("FAIL: no CSRF token on the page\n"); }

// 3. Submit with a marker you can find and delete later.
$marker = 'probe-' . date('YmdHi');
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query([
    '_token'  => $token,
    'name'    => $marker,
    'email'   => 'probe@example.com',
    'message' => 'automated probe, safe to delete',
]));
curl_exec($ch);

// 4. Assert the effect, not the response.
$found = (int) $pdo->query(
    "SELECT COUNT(*) FROM leads WHERE name = " . $pdo->quote($marker)
)->fetchColumn();

echo $found === 1 ? "OK\n" : "FAIL: submitted, nothing stored\n";

Step 4 is the point of the whole thing. Not "did the POST return 200" — did a row appear. That single assertion catches the frozen-token case, the silently failing mail transport, the validation rule someone tightened, and the database column that quietly truncated the input.

Give the probe an obvious marker and delete its rows on a schedule. Yes, this writes to production. That is the trade: a handful of tagged test rows a day against not knowing your lead form is dead.

Why this keeps happening

Three reasons, and none of them are laziness.

Monitoring defaults are shaped by hosting, not by the product. A default check answers "is the machine up and serving". That was the right question when a machine going down was the common failure. The common failure now is code that is up and wrong.

Green is a feeling, and feelings are expensive to give up. An unbroken status bar makes people stop looking. Silence should mean "the assertions passed". In practice it means "nothing asserted anything".

Silent failure has no symptom by definition. A crash gets escalated in an hour. A form that swallows submissions gets noticed when someone asks why the pipeline is empty — which, in our case, took three days, and only because someone went looking for a different number.

The short version

A status code is a transport fact. It says the request reached something. It does not say the page did its job.
Check every route, not the homepage. Your sitemap already lists them.
Assert one meaningful string per route, and grep the body for error signatures while you are there.
Diff the error log across every deploy, automatically. Cheapest check here.
Round-trip the forms that make you money, and assert the stored row, not the response.
Anything that renders "sometimes" — structured data, widgets, conditional blocks — needs an output assertion, because code review will not catch a function that is never called.

None of this is sophisticated. The whole set is a cron job and about a hundred lines. The reason to write it is that the alternative is finding out from a person, and by then you have already paid for the traffic.

I build and maintain news and e-commerce platforms at alestaweb.com. Most of what I know about monitoring came from outages exactly this dumb.

The Query That Ate 75% of Our Database CPU: A MySQL Full-Text Post-Mortem

Mahmut Gündüzalp — Thu, 06 Aug 2026 22:11:21 +0000

A news platform I maintain started serving pages in six seconds. Load average sat at 16 on a 12-core box for hours. Nothing had been deployed. Traffic was up, but not 10x up.

The cause turned out to be a single SELECT that looked completely reasonable — the kind of query that passes code review, works fine on a 5,000-row table, and quietly becomes a wrecking ball at 80,000 rows.

Here is the whole investigation: how I found it, why it was slow, what the fix was, and the three unrelated things I learned along the way.

Symptom first

The obvious metrics:

Load average 16.7 on 12 cores
MySQL at 92% CPU, resident memory 9.8 GB out of 15 GB
4.8 GB pushed into swap
Time to first byte on article pages: 6.3 seconds

The tempting move here is to start tuning. Bump the buffer pool, add more cache, blame the bots. I've done that before and it's mostly a way of not finding the bug.

Find the query, not a query

The mistake I see most often at this stage is taking one snapshot of the process list, spotting something slow, and declaring victory. One snapshot tells you what was running at one instant. It doesn't tell you what dominates.

Sample it instead:

for i in $(seq 1 20); do
  mysql -N -e "SELECT info FROM information_schema.processlist
               WHERE command='Execute' AND info IS NOT NULL"
done | sed -E 's/.*MATCH.*/RELATED-ARTICLES/' | sort | uniq -c | sort -rn

Twenty cheap samples, bucketed by shape. The result:

    202 RELATED-ARTICLES
     41 other article queries
     21 rate-limit counter

202 of 268 active queries — 75% — were the same statement. That's not a slow query problem, that's a "one feature is eating the server" problem.

The query

It powers the "related articles" block under every story:

SELECT a.id, a.title, a.summary, a.image, a.published_at,
       MATCH(a.title, a.summary) AGAINST(? IN NATURAL LANGUAGE MODE) AS relevance
FROM articles a
LEFT JOIN categories c ON a.category_id = c.id
WHERE MATCH(a.title, a.summary) AGAINST(? IN NATURAL LANGUAGE MODE)
  AND a.id != ? AND a.status = '1'
ORDER BY (relevance / (DATEDIFF(NOW(), COALESCE(a.sort_date, a.published_at)) / 30 + 1)) DESC
LIMIT 6

Read that ORDER BY again. It divides the relevance score by the article's age in months, so a strong match from 2019 loses to a decent match from last week. It's a genuinely nice ranking idea. Freshness matters in news.

And the search term passed in? The current article's entire title plus the first 200 characters of its summary.

Why it's expensive

Two things compound.

Natural language mode is not a filter, it's a scorer. Feed it a long string and it matches on any meaningful token in that string. I measured it:

SELECT COUNT(*) FROM articles
WHERE MATCH(title, summary) AGAINST('<full title + 200 chars of summary>'
                                    IN NATURAL LANGUAGE MODE);
-- 16,141

16,141 rows matched out of 80,836 — about 20% of the table. A long natural-language term is a very wide net. Every extra common word widens it further.

The ORDER BY cannot use an index. It sorts on an expression computed per row. The optimizer has no choice:

+----------+----------------+----------------------------------------------+
| type     | key            | Extra                                        |
+----------+----------------+----------------------------------------------+
| fulltext | ft_title_summ  | Using where; Using temporary; Using filesort  |
+----------+----------------+----------------------------------------------+

Using temporary; Using filesort on a fulltext scan is the whole story. To return 6 rows, the server:

scores ~16,000 rows,
materialises them into a temporary table,
computes relevance / (age/30 + 1) for every one of them,
sorts all 16,000,
throws away 15,994.

Measured cost: ~0.22 s of CPU per call. On every article page view. With half the traffic coming from crawlers walking the archive, each hitting a different article — so per-article caching had a near-zero hit rate for exactly the traffic causing the load.

At a handful of article views per second, that one query alone wants more cores than the machine has.

Fix 1: stop searching with a paragraph

The search term should be the subject of the article, not the article. I extract meaningful words from the title only — drop stopwords, drop anything under three characters, cap at eight words:

function searchTermFromTitle(string $title): string
{
    $stop = ['and','the','with','for','from','that','this','was','are','has'];
    $clean = preg_replace('/[^\p{L}\p{N}\s]+/u', ' ', $title);
    $clean = trim(preg_replace('/\s+/u', ' ', $clean));

    $picked = [];
    foreach (explode(' ', $clean) as $word) {
        $lower = mb_strtolower($word, 'UTF-8');
        if (mb_strlen($lower, 'UTF-8') < 3 || in_array($lower, $stop, true)) {
            continue;
        }
        $picked[] = $word;
        if (count($picked) >= 8) break;
    }
    return $picked ? implode(' ', $picked) : $clean;
}

Match count dropped from ~16,000–58,000 (it varied wildly by article) to ~2,500–13,000. Better, not solved.

A note for non-English text: build the stopword list for the language you actually store. Mechanically reusing an English list on Turkish, German or Finnish content will either strip nothing or strip the wrong things, and morphology means a naive list misses inflected forms.

Fix 2: rank in two stages

The real problem isn't the match count, it's sorting all matches by an expression. So don't. Take the top N by raw relevance — which the fulltext index can drive with a bounded sort — then apply the freshness weighting to those N in an outer query:

SELECT t.id, t.title, t.summary, t.image, t.published_at, t.relevance
FROM (
    SELECT a.id, a.title, a.summary, a.image, a.published_at,
           MATCH(a.title, a.summary) AGAINST(? IN NATURAL LANGUAGE MODE) AS relevance,
           COALESCE(a.sort_date, a.published_at) AS rank_date
    FROM articles a
    LEFT JOIN categories c ON a.category_id = c.id
    WHERE MATCH(a.title, a.summary) AGAINST(? IN NATURAL LANGUAGE MODE)
      AND a.id != ? AND a.status = '1'
    ORDER BY relevance DESC
    LIMIT 60
) t
ORDER BY (t.relevance / (DATEDIFF(NOW(), t.rank_date) / 30 + 1)) DESC
LIMIT 6

The expensive sort now runs over 60 rows instead of 16,000. The derived table has a LIMIT, so it can't be merged away — it's materialised, which is precisely what I want here.

Ranking quality barely moves. A result that wins after freshness weighting was already a strong relevance match; it was never sitting at position 4,000.

Results

Measured against real rows, old query vs new, five articles:

	Before	After
5 queries, total	2.571 s	0.572 s
Share of active queries	75%	16%
Load average	16.7	7.4
TTFB, worst page	6.34 s	1.91 s

4.5x on the query. And no caching layer involved — this is the same work, arranged so the database isn't asked to sort a haystack to hand back six needles.

Three things I learned that weren't the bug

InnoDB does not give space back. I pruned about 2.9 million rows from some log tables. data_free went up by 270 MB and the files didn't shrink. Deleted rows leave free pages inside the tablespace, reusable by that table but not returned to the OS. OPTIMIZE TABLE rebuilds it. If your cleanup job only deletes, your disk usage will never reflect it.

A dynamic variable can fail silently. innodb_buffer_pool_size is dynamic on modern MariaDB, so I resized it live. No error. The value didn't change. The reason: innodb_buffer_pool_chunk_size read as 0, and resizing works in chunk units. Always read the variable back after you set it — don't trust the absence of an error.

Long-lived MySQL processes bloat. That instance had been up 85 days with a 1 GB buffer pool and was holding 9.8 GB resident, 4.8 GB of it swapped. After a restart with a properly sized pool: 1.96 GB resident, swap essentially empty, and 8 GB handed back to the OS. Per-connection buffers and temp-table churn accumulate; a restart is sometimes the honest fix.

What I'd take away

Sample the process list, don't snapshot it. The dominant query and the slowest query are usually different queries, and the dominant one is what's hurting you.
Using temporary; Using filesort next to a LIMIT 6 is a smell. It means the server built the whole set to return a fraction of it.
Natural language full-text search scales with term length. A search term built from a whole paragraph is a query against a fifth of your table.
Fix the shape before you add cache. Caching this would have hidden it from users while the crawler traffic — cache-missing by construction — kept the CPU pinned.

The bug had been there for a long time. It only became visible when the archive got big enough and the crawlers got busy enough for those two curves to cross.

I build news and e-commerce platforms in PHP at Alesta WEB, an independent software company running since 2005.

Self-Hosted MCP: Building a Model Context Protocol Server in PHP

Mahmut Gündüzalp — Thu, 23 Jul 2026 11:11:26 +0000

Most of the Model Context Protocol tutorials you'll find are written in TypeScript or Python, because those are the languages with official SDKs. That leaves a fair number of us — people maintaining PHP applications that hold years of business data — wondering whether the protocol is even available to us.

It is. MCP is a wire protocol, not a library. If your language can read a line from standard input and write JSON back, you can implement a server in it. This post walks through what the protocol actually asks of you, what a PHP implementation looks like in outline, and — the part I underestimated — what changes when you expose an internal system to a model.

What MCP actually is

The Model Context Protocol is an open standard for connecting AI assistants to external systems: your data, your tools, your APIs. The problem it solves is combinatorial. Before a standard existed, every assistant needed a bespoke integration with every data source. MCP defines one interface so that any compliant client can talk to any compliant server.

Underneath, it's JSON-RPC 2.0. Requests carry a jsonrpc version, a method, a params object, and an id; responses carry the matching id and either a result or an error. Notifications are requests without an id and expect no reply. If you've implemented a JSON-RPC service before, you already know 80% of the transport story.

A server exposes three kinds of primitive:

Tools — actions the model can invoke. Each has a name, a description, and a JSON Schema describing its inputs. This is the primitive that does something: query a database, create a record, send a request. The model chooses when to call them.
Resources — data the model can read, addressed by URI. Files, records, generated documents. These are for context, not action, and the client generally decides what to pull in.
Prompts — reusable prompt templates the user can invoke deliberately, often surfaced in the client's UI as a slash command or menu item.

The distinction between tools and resources matters more than it first appears. A rough rule: tools are model-controlled, resources are application-controlled. If the model should decide whether to fetch something, make it a tool. If the user or the host application decides, make it a resource.

The two transports

MCP defines two standard transports, and picking the right one is the first architectural decision.

stdio. The client launches your server as a subprocess and talks to it over standard input and output — newline-delimited JSON, one message per line. This is the simplest possible setup: no ports, no HTTP server, no authentication layer, because the only thing that can talk to your process is the parent that spawned it. It's the right choice for anything running on the same machine as the client.

Two rules with stdio, both easy to break in PHP:

Never write anything but protocol messages to stdout. A stray echo, a var_dump left in from debugging, or a PHP warning printed to stdout corrupts the message stream and the client will fail to parse it. Send diagnostics to stderr instead, which the client typically forwards to a log.
Turn off output buffering and flush after every write, or your responses sit in a buffer while the client waits.

Streamable HTTP. The server runs as an ordinary HTTP endpoint. The client POSTs JSON-RPC messages to it; the server replies with either a single JSON response or a stream of server-sent events when it needs to push multiple messages for one request. This is the transport for a server that runs somewhere other than the user's machine — which, for most PHP shops, is the interesting case, because that's the deployment model we already know how to operate.

(An older HTTP+SSE transport exists in earlier revisions of the spec. New work should target Streamable HTTP.)

The handshake

Whichever transport you pick, the conversation starts the same way. The client sends initialize with the protocol version it speaks and the capabilities it supports. Your server replies with its own protocol version, its capabilities, and its name and version. The client then sends an initialized notification, and normal operation begins.

Capabilities are how the two sides negotiate. If your server doesn't implement resources, you don't advertise the resources capability, and a well-behaved client won't call resources/list. Don't advertise what you haven't built.

After the handshake, the methods that matter are predictable:

Method	What it does
`tools/list`	Return the tools you expose, with descriptions and input schemas
`tools/call`	Execute one tool with the given arguments, return its result
`resources/list`	Return available resources with their URIs
`resources/read`	Return the contents of one resource
`prompts/list`	Return available prompt templates
`prompts/get`	Return one filled-in prompt

A minimal but genuinely useful server is initialize plus tools/list plus tools/call. Everything else is optional.

What the PHP side looks like

The structural shape, transport-independent:

read a JSON-RPC message
  → dispatch on `method`
  → build a result (or an error)
  → write the response with the same `id`

For stdio, that's a loop over fgets(STDIN), json_decode, a match on the method name, and fwrite(STDOUT, json_encode($response) . "\n"). For Streamable HTTP, it's a single endpoint that decodes the request body and returns the encoded response. The dispatch layer in the middle is identical; only the read and write ends change. Write it that way from the start and you can support both from one codebase.

Three PHP-specific things worth knowing before you start:

Tool schemas. Every tool needs a JSON Schema for its inputs. Hand-writing those as nested arrays gets tedious fast. Deriving them from something you already maintain — a validation ruleset, a DTO, a set of typed constructor parameters read via reflection — keeps the schema and the actual implementation from drifting apart. Schema drift is the single most common cause of "the model keeps calling the tool wrong."

Error handling. Distinguish two kinds of failure. A malformed request or an unknown method is a protocol error: return a JSON-RPC error object. A tool that ran but failed — record not found, validation rejected the input — is a tool error: return a normal result with the error flag set and a human-readable message. The distinction matters because the second kind goes back to the model, which can read the message and adjust. A protocol error just tells it something broke. Convert PHP exceptions into the second kind wherever the failure is something the model could plausibly recover from.

Long-running work. PHP's request-per-process model is a good fit for stdio (the process lives as long as the session) and a slightly awkward one for HTTP if a tool takes minutes. Keep tool calls short. If a tool kicks off something slow, return a job identifier immediately and expose a second tool that reports status. Models handle that pattern well; they handle a request that times out badly.

Connecting it to Claude

Once the server runs, there are three broad ways to reach it.

Local, over stdio. Desktop and CLI clients — Claude Desktop, Claude Code, and various IDE integrations — let you register a server by specifying a command to run and its arguments (php, plus the path to your server script), and they manage the subprocess for you. This is the fastest way to get from "it responds to tools/list" to "I'm using it."

Remote, over HTTP. A Streamable HTTP server deployed at a URL can be registered with clients that support remote connections. The Claude API also has an MCP connector: you declare the server's URL in the request, and the API makes the connection server-side, so the model can call your tools without you writing a client loop. Note that hosted MCP servers generally authenticate with OAuth bearer tokens rather than a service's own native API key — those are different auth systems, and assuming the latter works is a common early stumble.

From your own code. If you're building an agent rather than using an existing client, most AI SDKs can convert MCP tool definitions into their native tool format, so an MCP server becomes a source of tools for a loop you control.

Whichever route, debug over stdio first. The failure modes are simpler: no TLS, no auth, no proxy, no CORS. Get the protocol right, then move it to HTTP.

The part I underestimated: exposure

Here's the thing that changes when you put an MCP server in front of an existing system. Every tool you expose is a capability granted to a model that is, at least in part, steered by text it reads from the outside world. If the model can be persuaded to call your tool, your tool runs.

The practical consequences:

Scope tools narrowly. A generic run_query tool that accepts arbitrary SQL is the most convenient thing to build and the worst thing to ship. Prefer specific tools with typed parameters — find_customer_by_email, list_orders_in_range — and validate every argument server-side as if it came from an anonymous HTTP request. It did, effectively.

Read and write are different risk classes. Read-only tools have a disclosure risk. Write tools have a this actually happened risk. Separate them, and put anything destructive or irreversible behind an explicit confirmation in the host application rather than trusting the model to be careful.

The description is part of the security surface. Tool descriptions are instructions the model reads. A vague description invites the model to call a tool in situations you didn't intend. Being prescriptive about when a tool should be used — not just what it does — measurably improves both correctness and safety.

Don't leak what you didn't mean to expose. Return the fields the task needs, not whole records. An internal note, a cost price, or a personal phone number that shouldn't be in a customer-facing answer shouldn't be in a tool result either. Filter at the server, not in the prompt.

Log every call. Tool name, arguments, caller, outcome. When someone asks "why did it do that," the log is the only answer you'll have.

None of this is exotic — it's the same discipline you'd apply to a public API endpoint. The difference is that the caller is a language model rather than a developer reading your docs, so ambiguity gets resolved in ways you didn't anticipate rather than surfacing as a support ticket.

Worth it?

For a PHP application sitting on years of accumulated business data, MCP is the cheapest bridge I've found between that data and an assistant that can actually reason about it. There's no SDK to wait for. It's JSON-RPC over a pipe or an HTTP endpoint — both things PHP has done well for twenty years.

Start with three read-only tools that answer questions someone in your organization asks weekly. Ship it over stdio to one person. See what they actually ask for. That's a far better spec than anything you'd design up front.

If you've built an MCP server in a language without an official SDK, I'd be curious what tripped you up — the transport, the schemas, or the scoping.

From Generic CMS to Purpose-Built: What 200 Site Migrations Taught Me

Mahmut Gündüzalp — Thu, 23 Jul 2026 11:09:31 +0000

Migrating a live website to a new content management system is one of those jobs that looks like a data transfer and turns out to be an exercise in risk management. The database part is usually the easy half. The hard half is that a working site has years of accumulated meaning attached to it — URLs search engines memorised, editors' habits, a content model that drifted far from whatever it was on day one.

Over the past years I've moved a few hundred sites — mostly news portals and online stores — off general-purpose, plugin-assembled CMS setups onto purpose-built platforms. Some of those migrations were clean. A few taught me things the hard way. This is the accumulated checklist, aimed at anyone about to do their first serious one.

Why sites migrate at all

Nobody migrates a CMS for fun. The trigger is almost always one of four things, and it's worth naming yours honestly before you start, because it determines what "success" means.

Plugin sprawl. A site starts generic and grows domain-specific through add-ons: one plugin for the ticker, one for the photo gallery, one for the ad slots, one for the newsletter, three for SEO. Each one is maintained by a different party on a different release cadence. The site works, but nobody can predict what a routine update will break.

Performance under spikes. A general CMS with fifteen plugins executes a lot of code to render a page that is, semantically, one article. For a news site where a single story can bring 40× normal traffic in ten minutes, that overhead stops being academic.

Editorial friction. This is the one clients articulate least well and feel most. If publishing a story with a photo gallery, a source credit, a related-articles block, and an agency attribution takes eleven clicks across four screens, your newsroom will invent workarounds, and your data model will slowly fill with those workarounds.

Cost of ownership. Ten commercial add-on licences plus a maintenance retainer, renewed annually, forever.

Write your trigger down. If it's performance, you need before/after numbers. If it's editorial friction, you need to time the actual publishing workflow before and after. A migration that can't be measured will be judged on vibes, and vibes after a migration are always bad for the first two weeks.

The part everyone underestimates: URLs

Here is the single highest-risk element of any migration of an established site: the URL is not a detail, it's the asset.

A news site that has been publishing for eight years has tens of thousands of indexed URLs. Some carry external links. Some rank on page one for terms the business depends on. Every one of them that returns a 404 after cutover is value thrown away, and search engines are considerably slower to give it back than they were to take it.

So before touching anything else:

Export the complete URL inventory. Not the sitemap — the sitemap is what the old CMS thinks exists. Pull the real list from server access logs (every distinct path with a 200 in the last 12 months), from the search console's indexed-pages report, and from the database. Union all three. The gap between the sitemap and the logs is routinely 20–30%, and it's where the old, still-ranking content lives.
Classify it. Articles, category pages, tag pages, author pages, static pages, media files, feeds, pagination, and the long tail of oddities — print views, AMP variants, tracking-parameter duplicates, dated archive pages.
Decide the target URL scheme once, on purpose. If the new system can preserve the existing pattern, preserve it. Redesigning URLs "while we're in there" doubles the risk of the migration for aesthetic gain. If you genuinely must change the scheme, you now owe every old URL a permanent redirect.
Build the redirect map as data, not as rules. Regex rules feel elegant and fail silently on the exceptions. A generated lookup table of old path → new path, produced from the actual inventory, is boring and verifiable. You can test all 40,000 entries automatically; you cannot test a regex against the URLs you forgot existed.
Verify with a crawl, not a spot check. After cutover, crawl the full old inventory against the live site and assert: status is 301 (not 302, not 200-with-different-content), the destination resolves 200 in one hop, and no chains or loops exist. Redirect chains are where migrations quietly leak ranking.

The rule I'd tattoo on a junior developer: a migration is not done when the content is moved; it's done when the old URLs still work.

Content models never map one-to-one

The second discovery of every migration is that the old content model is not what the documentation says. It's what editors made it.

You will find, reliably:

A "category" that is actually used as a workflow flag ("Ready", "Legal check").
Three tags that mean the same thing with different capitalisation, plus one with a trailing space.
HTML pasted into a plain-text summary field, because the summary needed a line break in 2019.
Images referenced by absolute URL to the old domain, inside article bodies.
A custom field that is empty on 90% of records and load-bearing on the other 10%.

Two practical responses. First, profile before you map. For every field you intend to migrate, run the actual distribution: how many distinct values, how many nulls, the longest value, how many contain HTML, how many contain absolute URLs. Ten minutes of profiling prevents a mapping decision built on an assumption.

Second, decide explicitly what not to migrate. Migration is the one moment when dropping dead weight is cheap. Fifteen years of unused tags, an abandoned events module, a decade of spam comments — carrying them forward means maintaining them forever. Just make the decision consciously and record it, rather than discovering the loss in month three.

The transformation itself I'd keep as a repeatable, re-runnable script, never a hand-edited dump:

read source record
  → normalise (trim, fix encoding, resolve relative media paths)
  → map fields to target model
  → validate (required fields present, slug unique, dates sane)
  → write, or write to a rejects file with the reason

Re-runnable matters more than it sounds. You will run the migration many times — against a copy, into staging, into a rehearsal, and finally for real. If the process is a script that takes the source and produces the target from scratch, every run is identical and every fix is permanent. If it's a script plus "and then I manually fixed those 40 rows", you have no repeatable process, and the final run will differ from the rehearsal in ways you can't see.

Media is where the disk space and the surprises are

Content rows are small. Media is not, and it's messier:

Files referenced in article bodies by absolute URL — these break the moment the domain or path changes and must be rewritten during transformation.
Orphan files: uploaded, never referenced. Often 30–50% of the media directory.
Referenced files that don't exist. Someone cleaned up a folder in 2021.
Generated thumbnail variants in a naming scheme the new system doesn't share.
Filenames with non-ASCII characters and spaces, which behave differently across filesystems and web servers.

Before the migration, produce two lists: referenced-but-missing (fix or accept — these become broken images), and present-but-unreferenced (usually skip). Never regenerate thumbnails on first request under production traffic; pre-generate them during the migration, or your launch-day load test will be an unintentional one.

Rehearse the cutover, then rehearse it again

The cutover is a sequence of steps under time pressure, which is exactly the condition in which people improvise. So write it down as a runbook with times, owners, and — for each step — the answer to "how do we undo this?"

A workable shape:

T-7 days: full migration into staging with production data. Editors do real work in it for a week. This is the only reliable way to surface editorial-workflow problems, and it's the step most often skipped for schedule reasons.
T-2 days: dry run of the entire cutover against staging, stopwatch running. You now know how long the real one takes. It's always longer than the estimate.
T-1 day: DNS TTL lowered. Content freeze agreed with the newsroom, in writing, with a named person responsible.
T-0: final delta sync (only content created since the main import), switch, smoke tests, redirect crawl.
T+1 hour: the checks below.
T+7 days: daily monitoring of crawl errors and index coverage.

Pick your window by traffic, not by convention. For a news site the quiet hour is usually early morning local time — but check the actual analytics, and check the editorial calendar. Cutting over the night before a major scheduled news event is a self-inflicted wound.

Define the rollback before you need it. What's the trigger (error rate above X, homepage down more than Y minutes)? Who decides? How long does reverting take? A rollback plan that takes four hours isn't a rollback plan; it's a wish. In practice this usually means: keep the old stack running and untouched, switch at the DNS or proxy layer, and don't decommission anything for at least a fortnight.

What to check in the first hour, and the first month

Immediately after cutover, in this order:

Homepage and a sample of article pages return 200 with the right content.
The redirect crawl passes across the full old-URL inventory.
Feeds and the sitemap are valid and reachable; submit the new sitemap.
robots.txt is the production one. (Staging's Disallow: / reaching production is a classic, and it is genuinely expensive.)
Canonical tags and structured data render correctly on article pages.
Search: does querying an old, well-known headline return it?
Publishing works end to end — an editor publishes a real story with an image while you watch.
Analytics and any ad slots are firing.

Then over the following weeks, watch crawl errors (a slow rise in 404s means the URL inventory had gaps), index coverage (a drop means something is deindexing content — check canonicals and robots directives first), server response time under real traffic, and the numbers that justified the migration in the first place.

Expect a short dip in traffic. A one-to-two week wobble while search engines re-crawl is normal even on a clean migration. A sustained four-week decline is not a wobble; it's a bug, and it's nearly always URLs, canonicals, or robots directives — in that order of likelihood.

The things I'd tell myself before the first one

Preserve URLs. Everything else is negotiable.
Profile the real data before mapping it. The schema is a claim; the data is the fact.
Make the migration a re-runnable script, never a manual fix-up.
Rehearse with editors, not just with developers. They will find in one afternoon what a test suite won't find in a month.
Keep the old system alive and reversible for two weeks. It costs a little hosting and buys a lot of sleep.
Measure the thing you migrated for. Otherwise the only feedback you'll get is the first complaint.

None of this is exotic. A CMS migration is mostly the discipline to do unglamorous inventory work before the interesting part, and the honesty to define in advance what "it went wrong" looks like.

If you've done a large migration, I'm curious what bit you that isn't on this list — the failure modes seem to be endlessly inventive.

PHP 8.2 in 2026: Why It's Still the Best Choice for News & E-commerce CMS

Mahmut Gündüzalp — Thu, 02 Jul 2026 13:16:52 +0000

Every year someone announces that PHP is finished, and every year a large slice of the web keeps running on it — including most of the news portals and online stores I work with. That's not nostalgia. After building content and commerce systems for 200+ production sites, PHP 8.2 keeps winning the specific fight that matters for this domain: shipping a stable, fast, server-rendered site that a small team can maintain for years without a rewrite.

This isn't a "PHP is actually cool now" hype piece. It's the concrete reasons a boring, typed, request-per-page runtime is still the right default for news and e-commerce CMS work in 2026 — and the honest places where it isn't.

The workload, not the benchmark

News and commerce sites have a very particular shape:

Read-heavy, cache-friendly. Most visitors are anonymous and see the same article or product page. You want to render HTML on the server, cache it hard, and get out of the way.
Spiky. A breaking story or a campaign can multiply traffic in minutes. The runtime has to degrade gracefully, not fall over.
Long-lived. These sites run for 5–10 years. The team that maintains them in year 6 is rarely the team that built them in year 1.
SEO-critical. For news, Google News and fast Largest Contentful Paint aren't nice-to-haves; they're the business.

PHP's execution model fits this shape almost embarrassingly well. Each request starts clean, does its work, and dies. No long-lived process accumulating memory leaks, no shared mutable state to reason about across requests. For a page that reads from a database, renders a template, and returns HTML, "shared-nothing" is a feature, not a limitation.

What 8.2 actually gave us

The jump from PHP 5.x/7.x thinking to 8.2 changed how these codebases read. A few features carry most of the weight in practice.

readonly properties made value objects trustworthy. In a CMS you pass a lot of small immutable things around — a resolved article, a price with currency, a category node. Being able to say "this cannot change after construction" at the language level removes a whole category of "who mutated this?" bugs.

final class Money
{
    public function __construct(
        public readonly int $amount,      // minor units (kuruş/cents)
        public readonly string $currency, // 'TRY', 'USD'
    ) {}

    public function withVat(int $ratePercent): self
    {
        return new self(
            (int) round($this->amount * (100 + $ratePercent) / 100),
            $this->currency,
        );
    }
}

No setter, no accidental mutation three layers down, and money math stays in integer minor units where it belongs.

Enums (from 8.1, but they land fully in 8.2 codebases) replaced the pile of const STATUS_DRAFT = 0 integers that every legacy CMS drags around. An article status or an order state becomes a real type the IDE and the type checker understand:

enum OrderStatus: string
{
    case Pending  = 'pending';
    case Paid     = 'paid';
    case Shipped  = 'shipped';
    case Refunded = 'refunded';

    public function isFinal(): bool
    {
        return $this === self::Refunded || $this === self::Shipped;
    }
}

Constructor promotion + typed properties cut the ceremony that made older PHP feel heavy. A service class is now mostly signal:

final class ArticleRenderer
{
    public function __construct(
        private readonly TemplateEngine $view,
        private readonly CacheInterface $cache,
    ) {}
}

never return types, readonly classes, and stricter type coercion together mean the type checker catches far more before code ever reaches staging. On a long-lived codebase, that's the difference between refactoring with confidence and refactoring with prayer.

None of these are flashy. Collectively they turn PHP from "scripting language you tolerate" into "typed application language that happens to have the best deployment story on the web."

Server-rendered HTML is a competitive advantage again

For a few years the default answer to "how do I build the frontend?" was a JavaScript SPA. For a news article or a product page, that was almost always the wrong trade. You paid a bundle-size and complexity tax to re-implement, in the browser, the one thing the server already does perfectly: turn data into HTML.

PHP renders HTML on the server as its native act. Pair it with a template engine and a sprinkle of hypermedia-style JavaScript for the genuinely interactive bits (filters, infinite scroll, cart updates) and you get:

First paint that doesn't wait on a JS runtime — good for LCP, good for Google News.
HTML that's fully present for crawlers and AI scrapers without a headless-render step.
A frontend a backend developer can maintain, instead of a second full stack.

The industry rediscovering server-rendered HTML in 2024–2026 was, from a PHP seat, watching everyone walk back to where the language already stood.

Handling the spikes: cache, don't scale

The read-heavy, spiky profile has a well-worn answer in PHP land, and 8.2's execution model makes it clean:

Full-page cache for anonymous traffic. Most visitors on a breaking story are logged out and identical; serve them a cached HTML page and never touch the database.
Opcode + preloading so the framework itself isn't re-parsed on every request.
A cache layer that switches on under load rather than being always-on — normal traffic hits the database for freshness, a spike flips the site into aggressive caching automatically.

Because each request is isolated, there's no warm-up state to protect and no cross-request corruption to worry about when you throw a cache in front. The mental model stays simple, which is exactly what you want at 3 a.m. when a story goes viral.

The maintenance argument nobody puts on slides

The feature that keeps me choosing PHP 8.2 for client work isn't in any release note: you can hand the codebase to a different developer in year 4 and they can be productive in a week.

The request lifecycle is obvious. Request in, response out.
Deployment is rsync and a cache flush, not an orchestration diagram.
The hosting is everywhere and cheap, which matters enormously for regional news sites and small stores.
The type system now documents intent well enough that a newcomer can read a service class and know what it does.

For software that has to outlive its authors, that boringness is the whole point. A clever runtime that only its original author understands is a liability on a 10-year site.

Where PHP 8.2 is not the answer

Being honest keeps this credible. I don't reach for PHP when:

The workload is long-lived and stateful — a websocket server, a real-time collaboration backend, a streaming pipeline. Shared-nothing-per-request is the wrong shape there; that's a job for a long-running process in another runtime.
The product is fundamentally a rich client app — a design tool, an editor with heavy live interaction. That genuinely wants a real frontend stack.
You need CPU-bound number crunching. PHP will do it; it won't be the tool you're happy with.

News and e-commerce CMS work is none of those. It's data-in, HTML-out, cache-heavy, maintained-for-years. That's PHP's home field.

The 2026 verdict

Language choice is a maintenance decision disguised as a technical one. For content and commerce systems, the constraints that actually bite are: fast server-rendered pages, graceful behavior under spikes, cheap ubiquitous hosting, and a codebase a small team can still understand years later. PHP 8.2 — typed, readonly, enum-shaped, opcode-cached — hits every one of those without asking you to be clever.

That's why the sites I build for news and e-commerce still start from PHP 8.2, and why I expect them to still be running, and still be maintainable, long after the next "PHP is dead" post. If you want to see what a modern PHP news and e-commerce stack looks like in production, that's the whole business at alestaweb.com.

Pick the runtime you can still afford to maintain in year six. For this domain, that's still PHP.

DEV Community: Mahmut Gündüzalp

Green Build, Broken Site: Three Next.js 16 Failures You Only See in Production

1. The image library that compiled to nothing

Symptom

Why it only happened in production

Fix

The check that would have caught it

2. Uploads that 404 until the next restart

Symptom

Why

Fix

The check that would have caught it

3. Scheduled posts that published but never appeared

Symptom

Why

Fix

The check that would have caught it

What the three have in common

Two Proxies, One Comma, Every Page a 500

The shape of the deployment

Two header lines are one string

Where it breaks

The obvious fix, and the second failure

What actually fixed it

The secureCookie line is not optional

The neighbouring headers have the same problem

X-Forwarded-For: the last value is the CDN

Canonical redirects: read the first value, or the CDN's own signal

Checklist for "works on staging, 500 behind the CDN"

The general shape

Six Rows Deleted. Three Decrements Applied. No Error.

The query

The measurement

Why nobody catches it in testing

The fixes that don't work

The fix

Finding the damage you already have

The general shape

The Socket Was Open. The Site Was Dead for Fourteen Hours.

What the checks reported

The log line that mattered

PPID 1

Why nothing restarted it

Fourteen hours

The fix, and it is boring

What to check instead

The short version

Lowercasing "İ" Adds a Character: How Turkish Text Broke a Keyword Router

The symptom

Trap 1: lowercasing a dotted capital I produces two code points

Unicode normalization will not save you

The other direction, for readers who do not write Turkish

Trap 2: unbounded substring matching

Trap 3: map order is a priority ranking you did not know you wrote

Normalize for matching, never for identity

How to verify, because you cannot eyeball this

The short version

The Word "null" Is Not a Null: How JSON Nulls Turn Into Text on Android

Why it does that

It also depends on which org.json you are compiled against

Where the null came from in the first place

Why the test accounts never caught it

Fix both layers, not one

Server side — normalize once, at the boundary

Client side — never call optString directly again

What about Gson, Moshi, kotlinx.serialization?

The five-minute audit

MySQL Will Quietly Eat Your Data If You Let It: Truncation, Charsets, and the 1366 You Never See

Part 1: The truncation

Finding the damage in a database you inherited

Part 2: Strict mode is necessary and not sufficient

Part 3: Charsets, where "it worked on the old server" lives

When you think data is corrupted, check the bytes

Part 4: The application-side version of the same bug

The checklist

Your Uptime Monitor Says 200. Your Contact Form Has Been Dead for Three Days.

What broke

Four ways to be broken and still return 200

What to assert instead

Check every route, not one route

The `secureCookie` line is not optional

`X-Forwarded-For`: the last value is the CDN

It also depends on which `org.json` you are compiled against

Client side — never call `optString` directly again