How I Built a Chaos Testing Harness for Video API Endpoints in PHP 8.4

At 02:40 UTC, the JP, KR and TW fetch workers each finished in under 900ms and wrote zero rows. Exit code 0. No alert. By morning, three of the eight regional feeds on TrendVidStream were serving a fourteen-hour-old trending list, and the FTS5 search index had drifted out of sync with the videos table because the delete pass had run and the insert pass had not.

Every unit test passed. Every integration test passed. The endpoint contract tests passed. What none of them covered was an upstream returning HTTP 200 with a well-formed JSON body containing an empty items array, because one API key out of three had its quota bucket drained mid-run.

That is the exact shape of the failure chaos testing exists to find. Not "the service is down" — your code already handles that, probably with a retry and a log line. The dangerous case is "the service is up and lying to you in a way your happy path treats as success."

This is the harness I built for it. Stack: PHP 8.4, SQLite with FTS5, cron-driven multi-region ingest across 8 regions, FTP-based deploy. That last constraint shapes everything, because when rollback means re-uploading a directory tree over FTP, you want your bugs found before the upload, not after.

Chaos testing a read-heavy, cron-fed video API

Most chaos engineering writing assumes Kubernetes, service meshes, and pods you can kill. I have none of that. What I have is:

• An ingest side: cron workers per region, calling an upstream video API, writing to SQLite, rebuilding FTS5 rows.
• A read side: /watch/{id}, /category/{slug}, /search?q=, all served from the same SQLite file, all cached at three layers.
• A deploy side: FTP, which is slow, non-atomic, and has no health-gated rollout.

So my fault domain isn't "node loss." It's:

• Upstream returns 200 with an empty or truncated payload
• Upstream returns 429 for one API key but not the others
• Upstream returns items with dates in the future or the distant past
• SQLite returns SQLITE_BUSY because a read query held a lock past the write window
• The worker is killed mid-transaction (cron timeout, LiteSpeed 180s ceiling, OOM)
• The FTS5 shadow table and the source table disagree
• One region succeeds and seven fail, or vice versa

The harness injects each of those deliberately, then asserts a set of invariants — properties that must hold no matter which faults fired. That distinction is the whole design. Traditional tests assert outputs for known inputs. A chaos harness asserts properties for unknown inputs.

The fault injection layer

Fault injection has to live at a seam you actually control. For me that's the HTTP client interface every ingest worker depends on. I wrap it in a decorator that reads a plan from the environment, so production code has zero chaos awareness — the wiring happens in the composition root and only when CHAOS_PLAN is set.

PHP 8.4 makes this pleasant: enums for the fault taxonomy, readonly classes for the plan, and \Random\Randomizer with an explicit engine so every run is reproducible from a seed.

<?php
declare(strict_types=1);

namespace App\Chaos;

enum Fault: string
{
    case Latency   = 'latency';
    case EmptyOk   = 'empty_ok';
    case Truncated = 'truncated';
    case Http429   = 'http_429';
    case Http503   = 'http_503';
    case ConnReset = 'conn_reset';
    case ClockSkew = 'clock_skew';
}

final readonly class ChaosPlan
{
    /** @param array<string,float> $faults fault value => probability */
    public function __construct(
        public array $faults,
        public int $seed,
        public array $scope = [],
    ) {}

    public static function fromEnv(): ?self
    {
        $raw = getenv('CHAOS_PLAN');
        if ($raw === false || $raw === '') {
            return null;
        }
        $cfg = json_decode($raw, true, flags: JSON_THROW_ON_ERROR);

        $faults = [];
        foreach ($cfg['faults'] ?? [] as $name => $p) {
            $faults[Fault::from($name)->value] = (float) $p;
        }

        return new self($faults, (int) ($cfg['seed'] ?? 1), $cfg['scope'] ?? []);
    }
}

final class ChaosHttpClient implements HttpClient
{
    private \Random\Randomizer $rng;

    public function __construct(
        private HttpClient $inner,
        private ChaosPlan $plan,
        private \Closure $log,
    ) {
        $this->rng = new \Random\Randomizer(
            new \Random\Engine\Xoshiro256StarStar($plan->seed)
        );
    }

    public function get(string $url, array $ctx = []): HttpResponse
    {
        if (!$this->inScope($ctx)) {
            return $this->inner->get($url, $ctx);
        }

        foreach ($this->plan->faults as $fault => $probability) {
            if ($this->rng->getFloat(0.0, 1.0) >= $probability) {
                continue;
            }
            ($this->log)(['fault' => $fault, 'url' => $url, 'ctx' => $ctx]);

            return match (Fault::from($fault)) {
                Fault::Latency => $this->sleepThen(1_800_000, $url, $ctx),
                Fault::EmptyOk => new HttpResponse(200, json_encode([
                    'items' => [],
                    'pageInfo' => ['totalResults' => 0, 'resultsPerPage' => 50],
                ], JSON_THROW_ON_ERROR)),
                Fault::Truncated => new HttpResponse(
                    200,
                    substr($this->inner->get($url, $ctx)->body, 0, 512)
                ),
                Fault::Http429 => new HttpResponse(429,
                    '{"error":{"code":429,"message":"quotaExceeded"}}'),
                Fault::Http503 => new HttpResponse(503, ''),
                Fault::ConnReset => throw new TransportException('Connection reset by peer'),
                Fault::ClockSkew => $this->skewDates($this->inner->get($url, $ctx)),
            };
        }

        return $this->inner->get($url, $ctx);
    }

    private function inScope(array $ctx): bool
    {
        foreach ($this->plan->scope as $key => $want) {
            if (($ctx[$key] ?? null) !== $want) {
                return false;
            }
        }
        return true;
    }

    private function sleepThen(int $usec, string $url, array $ctx): HttpResponse
    {
        usleep($usec);
        return $this->inner->get($url, $ctx);
    }

    private function skewDates(HttpResponse $res): HttpResponse
    {
        $body = json_decode($res->body, true, flags: JSON_THROW_ON_ERROR);
        foreach ($body['items'] ?? [] as $i => $item) {
            if (isset($item['snippet']['publishedAt'])) {
                $body['items'][$i]['snippet']['publishedAt'] =
                    gmdate('c', time() + 86400 * 400);
            }
        }
        return new HttpResponse($res->status, json_encode($body, JSON_THROW_ON_ERROR));
    }
}

Three design notes that matter more than the code:

• The scope filter. ['region' => 'JP'] lets me fault exactly one region and leave seven healthy. Partial failure is far more interesting than total failure, because total failure usually trips an obvious guard.
• Seeded RNG. When a scenario finds a bug, I need to replay it byte-for-byte. Xoshiro256StarStar($seed) gives that. A failing run prints its seed, and re-running with the same seed and plan reproduces it.
• ClockSkew is not a network fault. It's a data fault, and it is the one that has found the most bugs. Video sorting, "trending in the last 24h" windows, and cache TTL math all quietly assume publishedAt <= now.

Invariants beat assertions

The part people skip. If your chaos run only checks "did it exit 0," you've built a very expensive smoke test. The value is in properties that must hold across every scenario.

Mine live in one class, run after every scenario, and return a list of violated property names rather than throwing on the first failure — I want the full picture from one run.

<?php
declare(strict_types=1);

namespace App\Chaos;

final class Invariants
{
    private const MAX_STALE_CYCLES = 3;

    public function __construct(
        private \PDO $db,
        private int $cycleSeconds,
    ) {}

    /** @return list<string> */
    public function check(): array
    {
        $violations = [];

        // I1: FTS5 shadow table must match the live row count exactly.
        $drift = (int) $this->db->query(
            'SELECT (SELECT COUNT(*) FROM videos WHERE deleted_at IS NULL)
                  - (SELECT COUNT(*) FROM videos_fts)'
        )->fetchColumn();
        if ($drift !== 0) {
            $violations[] = "I1_fts_drift:{$drift}";
        }

        // I2: every stored video must be resolvable through search.
        $orphans = (int) $this->db->query(
            "SELECT COUNT(*) FROM videos v
             WHERE v.deleted_at IS NULL
               AND NOT EXISTS (SELECT 1 FROM videos_fts f WHERE f.rowid = v.id)"
        )->fetchColumn();
        if ($orphans > 0) {
            $violations[] = "I2_unsearchable_rows:{$orphans}";
        }

        // I3: no region may be starved while its peers advance.
        $rows = $this->db->query(
            'SELECT region, MAX(fetched_at) AS last FROM videos GROUP BY region'
        )->fetchAll(\PDO::FETCH_ASSOC);
        $newest = max(array_map(static fn(array $r): int => (int) $r['last'], $rows ?: [['last' => 0]]));
        foreach ($rows as $r) {
            $lag = ($newest - (int) $r['last']) / $this->cycleSeconds;
            if ($lag > self::MAX_STALE_CYCLES) {
                $violations[] = sprintf('I3_region_starved:%s:%.1f', $r['region'], $lag);
            }
        }

        // I4: no future-dated content ever reaches the read side.
        $future = (int) $this->db->query(
            "SELECT COUNT(*) FROM videos WHERE published_at > strftime('%s','now') + 3600"
        )->fetchColumn();
        if ($future > 0) {
            $violations[] = "I4_future_published:{$future}";
        }

        // I5: a fetch cycle must never reduce the catalogue by more than 20%.
        $shrink = (float) $this->db->query(
            'SELECT CASE WHEN prev_count = 0 THEN 0.0
                         ELSE 1.0 - (CAST(cur_count AS REAL) / prev_count) END
             FROM fetch_audit ORDER BY id DESC LIMIT 1'
        )->fetchColumn();
        if ($shrink > 0.20) {
            $violations[] = sprintf('I5_catalogue_shrink:%.2f', $shrink);
        }

        return $violations;
    }
}

I5 is the one that would have caught my original incident. An empty-200 followed by a "replace the region's rows" write path deletes everything and inserts nothing. No exception, no non-zero exit. Just a catalogue that silently shrinks by one eighth. A single invariant on the write delta turns that from a next-morning discovery into a failed CI run.

fetch_audit is a two-column table the worker writes on every cycle: the row count before and after. It costs one INSERT per run and it's the highest-value table in the schema.

The scenario runner

The runner is Python, because I want the orchestration to be a separate process from the thing under test — if the worker hangs, wedges, or gets OOM-killed, the runner has to survive and record that as a result rather than dying with it.

It builds a matrix of scenarios, runs them with bounded concurrency against isolated database copies, and collects invariant violations.

#!/usr/bin/env python3
"""chaos.py - run a fault matrix against the regional ingest workers."""
import asyncio
import itertools
import json
import os
import shutil
import sys
import tempfile
from dataclasses import dataclass, asdict

REGIONS = ["US", "GB", "JP", "KR", "TW", "SG", "VN", "TH"]
FAULTS = ["empty_ok", "truncated", "http_429", "http_503", "conn_reset", "clock_skew"]
PROBABILITIES = [0.15, 0.60, 1.00]
GOLDEN_DB = "tests/fixtures/golden.sqlite"
TIMEOUT = 120


@dataclass(frozen=True)
class Scenario:
    fault: str
    probability: float
    region: str | None
    seed: int

    @property
    def name(self) -> str:
        scope = self.region or "all"
        return f"{self.fault}@{self.probability}:{scope}:s{self.seed}"

    def plan(self) -> str:
        scope = {"region": self.region} if self.region else {}
        return json.dumps({
            "faults": {self.fault: self.probability},
            "seed": self.seed,
            "scope": scope,
        })


def build_matrix(seed_base: int) -> list[Scenario]:
    scenarios = []
    for i, (fault, prob) in enumerate(itertools.product(FAULTS, PROBABILITIES)):
        # one blast-radius-of-one run, one blast-radius-of-all run
        scenarios.append(Scenario(fault, prob, REGIONS[i % len(REGIONS)], seed_base + i))
        scenarios.append(Scenario(fault, prob, None, seed_base + 1000 + i))
    return scenarios


async def run_one(sc: Scenario, sem: asyncio.Semaphore) -> dict:
    async with sem:
        workdir = tempfile.mkdtemp(prefix="chaos-")
        db = os.path.join(workdir, "app.sqlite")
        shutil.copy(GOLDEN_DB, db)

        env = {**os.environ, "CHAOS_PLAN": sc.plan(), "DB_PATH": db, "FETCH_REGIONS": ",".join(REGIONS)}
        proc = await asyncio.create_subprocess_exec(
            "php", "cron/fetch_videos.php",
            stdout=asyncio.subprocess.PIPE,
            stderr=asyncio.subprocess.PIPE,
            env=env,
        )
        try:
            out, err = await asyncio.wait_for(proc.communicate(), timeout=TIMEOUT)
            timed_out = False
        except asyncio.TimeoutError:
            proc.kill()
            out, err, timed_out = b"", b"killed", True

        check = await asyncio.create_subprocess_exec(
            "php", "tools/check_invariants.php", db,
            stdout=asyncio.subprocess.PIPE, env=env,
        )
        raw, _ = await check.communicate()
        violations = json.loads(raw or b"[]")

        shutil.rmtree(workdir, ignore_errors=True)
        return {
            **asdict(sc),
            "name": sc.name,
            "exit": proc.returncode,
            "timed_out": timed_out,
            "violations": violations,
            "stderr": err.decode()[-400:],
        }


async def main() -> int:
    seed_base = int(sys.argv[1]) if len(sys.argv) > 1 else 20260805
    sem = asyncio.Semaphore(int(os.getenv("CHAOS_CONCURRENCY", "4")))
    results = await asyncio.gather(*(run_one(s, sem) for s in build_matrix(seed_base)))

    failed = [r for r in results if r["violations"] or r["timed_out"]]
    print(json.dumps({"total": len(results), "failed": len(failed), "cases": failed}, indent=2))
    return 1 if failed else 0


if __name__ == "__main__":
    raise SystemExit(asyncio.run(main()))

A few things I got wrong the first time and fixed here:

• Each scenario gets its own database copy. Sharing one file means a violation in scenario 12 shows up in scenario 40 and you waste an afternoon.
• A timeout is a result, not a crash. asyncio.wait_for plus proc.kill() records "this scenario hung" as data. Hangs under fault injection are real bugs — usually a retry loop with no total-time budget.
• Concurrency is capped and configurable. SQLite under eight concurrent writers behaves differently than under four, and I want to control that variable rather than let CI's core count decide it.

Injecting SQLite and FTS5 faults

Network faults are the easy half. The failures that actually corrupted my search index came from the storage layer, and you can't inject those through an HTTP decorator.

Two techniques cover almost everything, and neither needs a mocking library.

Hold a real write lock. Start a second connection, open an exclusive transaction, sleep, roll back. The worker under test sees genuine SQLITE_BUSY from the real engine, with real timing:

<?php
// tools/chaos/hold_lock.php <db-path> <hold-ms>
declare(strict_types=1);

[$_, $path, $holdMs] = $argv + [null, null, '2000'];

$hold = new PDO('sqlite:' . $path, options: [
    PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
]);
$hold->exec('PRAGMA busy_timeout = 0');
$hold->exec('CREATE TABLE IF NOT EXISTS chaos_lock (ts INTEGER)');

$hold->beginTransaction();
$hold->exec('INSERT INTO chaos_lock (ts) VALUES (strftime(\'%s\', \'now\'))');

fwrite(STDERR, "holding write lock for {$holdMs}ms\n");
usleep((int) $holdMs * 1000);

$hold->rollBack();

Kill the worker mid-transaction. The runner sends SIGKILL at a randomised offset inside the write window. This is the only reliable way to find code that writes to videos and videos_fts in two separate transactions — a bug that is invisible until the process dies between them, and then permanently visible as index drift.

After every kill, the harness runs PRAGMA integrity_check and INSERT INTO videos_fts(videos_fts) VALUES('integrity-check'). The second one is FTS5-specific and worth knowing about: it validates the shadow tables against the content table and errors if they disagree. It caught two ordering bugs in my rebuild path that integrity_check alone was perfectly happy with.

One more storage-layer fault worth injecting: set PRAGMA max_page_count to just above the current database size before a run. The worker hits SQLITE_FULL mid-write without you needing to fill a disk. Cheap, deterministic, and it exposes every place you assumed an INSERT can't fail.

Read-side chaos under concurrent load

Ingest correctness is half the surface. The other half is what /search, /category and /watch return while a fault-injected fetch cycle is writing. My read path assumed a stable snapshot; SQLite in WAL mode mostly gives you that, but the page cache layer sitting in front of it does not.

A small Go driver hammers the read endpoints during each scenario and asserts response-level properties. Go here purely because I want thousands of concurrent requests without the driver itself becoming the bottleneck:

package main

import (
    "encoding/json"
    "flag"
    "fmt"
    "net/http"
    "os"
    "sync"
    "sync/atomic"
    "time"
)

type searchResp struct {
    Total int `json:"total"`
    Items []struct {
        ID          string `json:"id"`
        Title       string `json:"title"`
        PublishedAt int64  `json:"published_at"`
    } `json:"items"`
}

func main() {
    base := flag.String("base", "http://127.0.0.1:8080", "base url")
    workers := flag.Int("workers", 32, "concurrent readers")
    dur := flag.Duration("for", 60*time.Second, "run duration")
    flag.Parse()

    queries := []string{"trending", "documentary", "live", "k-pop", "highlights"}
    var reqs, violations int64
    deadline := time.Now().Add(*dur)
    client := &http.Client{Timeout: 5 * time.Second}

    var wg sync.WaitGroup
    for w := 0; w < *workers; w++ {
        wg.Add(1)
        go func(w int) {
            defer wg.Done()
            var lastTotal = -1
            for time.Now().Before(deadline) {
                q := queries[w%len(queries)]
                res, err := client.Get(fmt.Sprintf("%s/search?q=%s", *base, q))
                if err != nil {
                    atomic.AddInt64(&violations, 1)
                    continue
                }
                var body searchResp
                dec := json.NewDecoder(res.Body)
                decErr := dec.Decode(&body)
                res.Body.Close()
                atomic.AddInt64(&reqs, 1)

                switch {
                case res.StatusCode != 200:
                    fmt.Fprintf(os.Stderr, "R1 status=%d q=%s\n", res.StatusCode, q)
                    atomic.AddInt64(&violations, 1)
                case decErr != nil:
                    fmt.Fprintf(os.Stderr, "R2 malformed json q=%s\n", q)
                    atomic.AddInt64(&violations, 1)
                case lastTotal >= 0 && body.Total < lastTotal/2:
                    // R3: result count must not halve between two reads
                    fmt.Fprintf(os.Stderr, "R3 collapse %d -> %d q=%s\n", lastTotal, body.Total, q)
                    atomic.AddInt64(&violations, 1)
                }
                if decErr == nil {
                    lastTotal = body.Total
                }
            }
        }(w)
    }
    wg.Wait()

    fmt.Printf("requests=%d violations=%d\n", reqs, violations)
    if violations > 0 {
        os.Exit(1)
    }
}

R3 — the result-count collapse check — is deliberately loose. It doesn't care about exact counts, which change legitimately during ingest. It only fires when search results halve between two consecutive reads, which never happens legitimately and always means the FTS5 rebuild is visible mid-flight.

What it actually found

Six weeks of running this on every push, plus a nightly full-matrix run:

• The empty-200 delete-then-insert bug. Fixed by making the region write a single transaction and refusing to commit when the incoming item count is under 20% of the previous cycle.
• A retry loop with no total budget. Under http_503 at probability 1.0, a worker retried with exponential backoff forever. It never crashed; it just ran past the cron interval and overlapped with the next invocation, and two workers writing the same region produced duplicate rows. Fixed with a wall-clock deadline and a per-region advisory lock.
• Future-dated content poisoning the trending sort. clock_skew surfaced this in the first run. A single item with publishedAt 400 days ahead pinned itself to the top of every region's feed.
• FTS5 drift after mid-transaction kills. The rebuild wrote the content table and the index in separate transactions. Now they're one.
• SQLITE_BUSY returning a 500 to real users. The read path had no busy_timeout set on its connection. Three-character fix, found by holding a lock for 2 seconds.

Five real bugs, four of which would have reached production and none of which any existing test caught. That's the return on maybe three days of harness work.

Rules that kept it useful

Chaos harnesses rot fast. These are the constraints that stopped mine from becoming a flaky test suite everyone ignores:

• Every failure prints its seed and plan. If you can't replay a failure in one command, engineers will re-run CI until it goes green. Reproducibility isn't a nicety here; it's what separates a harness from noise.
• Invariants live in application code, not test code. Invariants::check() also runs as a lightweight post-cron health check in production. The same properties guard both.
• Faults are opt-in by environment variable, always. No chaos class is reachable unless CHAOS_PLAN is set. Given an FTP deploy that ships whatever is in the directory, a fault injector that could activate by accident is worse than no fault injector.
• The matrix stays small enough to run on every push. Full matrix nightly, a six-scenario smoke subset per push. A harness that takes 40 minutes gets disabled within a month.
• New incident, new scenario. Every production surprise becomes a permanent entry in FAULTS. That's the ratchet — the harness only gets better at catching the things that actually bite you.

Conclusion

The useful reframe is this: your tests check that correct inputs produce correct outputs. A chaos harness checks that incorrect inputs — malformed, empty, late, duplicated, half-written — never produce plausible-looking outputs. The second category is where multi-region video ingest breaks, because upstream APIs fail politely and cron swallows the evidence.

You don't need orchestration infrastructure to do this. A decorator around your HTTP client, a handful of invariant queries, a runner that copies the database per scenario, and the discipline to seed your RNG will get you most of the value. Start with three faults — empty-200, mid-transaction kill, and clock skew — and one invariant on write deltas. That combination alone would have saved me a fourteen-hour stale feed across three regions.

Write the invariant before you write the fault. The fault only tells you what happened; the invariant tells you why it was wrong.