At 3:11 AM one Tuesday, the /watch endpoint on our platform started returning 200s with empty bodies. No 500s, no error logs, nothing our alerting was watching for. The cause was mundane: an upstream metadata provider had started responding in 8–12 seconds instead of 200ms, our PHP-FPM workers piled up waiting on it, LiteSpeed's connection pool saturated, and Cloudflare began serving stale-or-empty cache entries to real users. Everything was technically "up." Everything was broken.
That incident is why I ended up building a chaos testing harness specifically for video API endpoints. I run DailyWatch, a free video discovery platform, and our read path is deceptively simple on paper: search, category listings, and per-video watch pages. But every one of those endpoints fans out to caches, a SQLite database with FTS5 full-text indexes, and one or more flaky upstream metadata sources. The interesting failures never live in a single function — they live in the seams between components under stress. Chaos testing is how you go looking for them on purpose, during business hours, instead of getting paged at 3 AM.
This post walks through the harness we actually run: how we model the failure surface, a fault-injection layer written in PHP 8.4, a Python driver that orchestrates scenarios, a circuit breaker that keeps the FTS5 read path alive when upstreams die, and — most importantly — how we assert on SLOs instead of eyeballing graphs.
Why video discovery endpoints fail in weird ways
A CRUD API mostly fails cleanly: the database is up or it isn't. Video discovery is different because a single response is assembled from sources with wildly different reliability profiles:
- The local hot path: SQLite FTS5 queries for search and category pages. Fast, in-process, almost never the problem — until the database file is locked during a cron-driven bulk insert.
- Cache layers: LiteSpeed page cache, a PHP file cache, and a data cache. Each can serve stale data, miss, or — the nasty one — return a partial object that looks valid.
- Upstream metadata: view counts, thumbnails, channel info. High latency variance, occasional 429s, and the occasional multi-second stall that never quite times out.
The failure modes that matter are combinations: a cache miss plus a slow upstream plus a concurrent write lock. You cannot find those by unit-testing components in isolation. You have to inject faults into a running system and watch what the whole thing does.
So the first job is to enumerate the failure surface honestly. I keep this as a plain data structure, because the harness reads it directly:
-
upstream_latency: add N ms of delay before returning -
upstream_error: return a 5xx or a 429 from a dependency -
upstream_partial: return a syntactically valid but incomplete payload -
db_lock: hold a write transaction to forceSQLITE_BUSY -
cache_poison: serve a truncated cache entry -
clock_skew: shift TTL math so cache entries expire early or never
Every scenario in the harness is one or more of these, applied at a chosen rate, against a chosen endpoint.
A fault-injection middleware in PHP 8.4
The cleanest place to inject faults is a middleware that wraps the request before it reaches the controller. I gate the whole thing behind an environment flag so it is physically impossible to enable in production, and behind a header so a test driver can target specific requests without affecting real traffic sharing the same staging box.
Here is the core injector. It reads a fault spec from a request header, decides probabilistically whether to fire, and mutates the request lifecycle accordingly.
<?php
declare(strict_types=1);
final class ChaosMiddleware
{
private const HEADER = 'X-Chaos-Spec';
public function __construct(
private readonly bool $enabled, // from CHAOS_ENABLED env
private readonly \Closure $rng, // injectable RNG: fn(): float in [0,1)
) {}
public function handle(array $server): ?ChaosAction
{
if (!$this->enabled) {
return null; // hard off outside staging
}
$raw = $server['HTTP_X_CHAOS_SPEC'] ?? '';
if ($raw === '') {
return null;
}
// Spec format: "fault=upstream_latency;rate=0.5;ms=6000"
$spec = $this->parse($raw);
$rate = (float) ($spec['rate'] ?? 1.0);
if (($this->rng)() >= $rate) {
return null; // this request drew clean
}
return match ($spec['fault'] ?? '') {
'upstream_latency' => ChaosAction::latency((int) ($spec['ms'] ?? 3000)),
'upstream_error' => ChaosAction::error((int) ($spec['code'] ?? 503)),
'upstream_partial' => ChaosAction::partial(),
'db_lock' => ChaosAction::dbLock((int) ($spec['ms'] ?? 2000)),
'cache_poison' => ChaosAction::cachePoison(),
default => null,
};
}
/** @return array<string,string> */
private function parse(string $raw): array
{
$out = [];
foreach (explode(';', $raw) as $pair) {
[$k, $v] = array_pad(explode('=', $pair, 2), 2, '');
$out[trim($k)] = trim($v);
}
return $out;
}
}
final class ChaosAction
{
private function __construct(
public readonly string $kind,
public readonly int $arg = 0,
) {}
public static function latency(int $ms): self { return new self('latency', $ms); }
public static function error(int $code): self { return new self('error', $code); }
public static function partial(): self { return new self('partial'); }
public static function dbLock(int $ms): self { return new self('db_lock', $ms); }
public static function cachePoison(): self { return new self('cache_poison'); }
public function applyUpstream(callable $realCall): mixed
{
return match ($this->kind) {
'latency' => (function () use ($realCall) {
usleep($this->arg * 1000);
return $realCall();
})(),
'error' => throw new UpstreamException($this->arg),
'partial' => array_slice((array) $realCall(), 0, 1),
default => $realCall(),
};
}
}
The important design choices here: faults are opt-in per request via a header, the RNG is injected so tests are deterministic when you want them to be, and the whole thing is a no-op unless CHAOS_ENABLED is set. On our stack that env flag only exists on the LiteSpeed staging vhost, never on the production one.
Keeping the FTS5 read path alive: a circuit breaker
Injecting faults is only useful if the endpoint is supposed to survive them. The single highest-value pattern for a discovery API is a circuit breaker in front of every non-local dependency, backed by a fast local fallback. When the upstream metadata provider is dying, we should still serve search results from FTS5 with whatever cached metadata we have, and degrade gracefully rather than block.
This is the breaker we wrap around upstream calls. It trips after a threshold of failures, stays open for a cooldown, then allows a single probe request through (half-open) before fully closing again.
<?php
declare(strict_types=1);
final class CircuitBreaker
{
private const CLOSED = 'closed';
private const OPEN = 'open';
private const HALF = 'half_open';
public function __construct(
private readonly \PDO $db, // SQLite state store
private readonly string $name,
private readonly int $threshold = 5,
private readonly int $cooldownSec = 30,
private readonly \Closure $now, // fn(): int -> unix ts, injectable
) {}
/**
* @param callable():T $call the risky upstream call
* @param callable():T $fallback the local/degraded path
* @template T
*/
public function run(callable $call, callable $fallback): mixed
{
$state = $this->load();
$now = ($this->now)();
if ($state['status'] === self::OPEN) {
if ($now - $state['opened_at'] < $this->cooldownSec) {
return $fallback(); // fail fast, do not touch the sick upstream
}
$this->transition(self::HALF, $state['failures'], $state['opened_at']);
}
try {
$result = $call();
$this->transition(self::CLOSED, 0, 0); // success resets everything
return $result;
} catch (\Throwable $e) {
$failures = $state['failures'] + 1;
if ($failures >= $this->threshold || $state['status'] === self::HALF) {
$this->transition(self::OPEN, $failures, $now);
} else {
$this->transition(self::CLOSED, $failures, 0);
}
return $fallback();
}
}
/** @return array{status:string,failures:int,opened_at:int} */
private function load(): array
{
$stmt = $this->db->prepare(
'SELECT status, failures, opened_at FROM circuit_state WHERE name = ?'
);
$stmt->execute([$this->name]);
$row = $stmt->fetch(\PDO::FETCH_ASSOC);
return $row ?: ['status' => self::CLOSED, 'failures' => 0, 'opened_at' => 0];
}
private function transition(string $status, int $failures, int $openedAt): void
{
$this->db->prepare(
'INSERT INTO circuit_state (name, status, failures, opened_at)
VALUES (:n, :s, :f, :o)
ON CONFLICT(name) DO UPDATE SET
status = :s, failures = :f, opened_at = :o'
)->execute([':n' => $this->name, ':s' => $status, ':f' => $failures, ':o' => $openedAt]);
}
}
And here is how the search controller uses it. The upstream enriches results with fresh view counts; the fallback serves the FTS5 hit list with whatever is already in the local table. A user searching for "documentary" during an upstream outage still gets relevant videos — just with slightly stale counts.
<?php
public function search(string $q, CircuitBreaker $breaker): array
{
// Local FTS5 query is the source of truth for *which* videos match.
$stmt = $this->db->prepare(
"SELECT v.id, v.title, v.channel, v.cached_views
FROM videos v
JOIN videos_fts f ON f.rowid = v.rowid
WHERE videos_fts MATCH :q
ORDER BY rank
LIMIT 40"
);
$stmt->execute([':q' => $this->sanitizeFtsQuery($q)]);
$hits = $stmt->fetchAll(\PDO::FETCH_ASSOC);
// Enrichment is best-effort and guarded by the breaker.
return $breaker->run(
call: fn() => $this->enrichWithLiveMetadata($hits), // risky upstream
fallback: fn() => $hits, // degraded but correct
);
}
The key insight for video discovery specifically: the ranking (from FTS5) must never depend on a network call. Metadata enrichment is decoration. Once you draw that line, most chaos scenarios become survivable by construction.
Driving chaos from the outside
The harness itself lives in Python because I want it to run against a real HTTP surface — through LiteSpeed and, in one profile, through Cloudflare — not against PHP objects in a test process. Each scenario is a small dataclass; the driver fires concurrent requests carrying the X-Chaos-Spec header and records latency and status for every one.
import asyncio
import time
from dataclasses import dataclass, field
import httpx
@dataclass
class Scenario:
name: str
endpoint: str
chaos_spec: str # e.g. "fault=upstream_latency;rate=0.5;ms=6000"
concurrency: int = 25
total_requests: int = 500
slo_p99_ms: float = 800.0
slo_success_rate: float = 0.99
@dataclass
class Result:
latencies_ms: list[float] = field(default_factory=list)
statuses: list[int] = field(default_factory=list)
errors: int = 0
async def _worker(client, scenario, result, sem, budget):
while True:
async with budget:
if budget.done:
return
async with sem:
start = time.perf_counter()
try:
r = await client.get(
scenario.endpoint,
headers={"X-Chaos-Spec": scenario.chaos_spec},
timeout=15.0,
)
result.latencies_ms.append((time.perf_counter() - start) * 1000)
result.statuses.append(r.status_code)
except (httpx.TimeoutException, httpx.TransportError):
result.latencies_ms.append((time.perf_counter() - start) * 1000)
result.errors += 1
class Budget:
"""Shared, thread-safe-ish request budget for the asyncio loop."""
def __init__(self, total: int):
self._remaining = total
self.done = False
async def __aenter__(self):
if self._remaining <= 0:
self.done = True
else:
self._remaining -= 1
return self
async def __aexit__(self, *exc):
return False
async def run_scenario(base_url: str, scenario: Scenario) -> Result:
result = Result()
budget = Budget(scenario.total_requests)
sem = asyncio.Semaphore(scenario.concurrency)
async with httpx.AsyncClient(base_url=base_url) as client:
workers = [
asyncio.create_task(_worker(client, scenario, result, sem, budget))
for _ in range(scenario.concurrency)
]
await asyncio.gather(*workers)
return result
A scenario file for us looks like a list of these dataclasses: baseline (no chaos), 50% upstream latency at 6s, 100% upstream 503, partial payloads at 30%, and a combined "bad night" scenario that layers db-lock and latency together. The combined ones are where the interesting bugs still hide.
Assert on SLOs, not on vibes
The part everyone skips is the assertion. It is not enough to run traffic and look at a dashboard — that is how regressions sneak back in. Every scenario declares its SLOs up front, and the harness fails the run (non-zero exit) if reality misses them. This is what turns chaos testing from a party trick into a CI gate.
import statistics
import sys
def percentile(values: list[float], p: float) -> float:
if not values:
return float("inf")
ordered = sorted(values)
k = (len(ordered) - 1) * p
lo = int(k)
hi = min(lo + 1, len(ordered) - 1)
return ordered[lo] + (ordered[hi] - ordered[lo]) * (k - lo)
def evaluate(scenario: Scenario, result: Result) -> bool:
total = len(result.statuses) + result.errors
# "success" for a discovery endpoint = 2xx AND a non-empty body contract.
# The driver records status; body-emptiness is checked server-side via a
# sentinel header the controller sets, collapsed here into status 599.
good = sum(1 for s in result.statuses if 200 <= s < 300 and s != 599)
success_rate = good / total if total else 0.0
p99 = percentile(result.latencies_ms, 0.99)
ok_rate = success_rate >= scenario.slo_success_rate
ok_p99 = p99 <= scenario.slo_p99_ms
print(f"[{scenario.name}] success={success_rate:.4f} "
f"(SLO {scenario.slo_success_rate}) p99={p99:.0f}ms "
f"(SLO {scenario.slo_p99_ms}ms) errors={result.errors}")
return ok_rate and ok_p99
async def main(base_url: str, scenarios: list[Scenario]) -> int:
failures = 0
for scenario in scenarios:
result = await run_scenario(base_url, scenario)
if not evaluate(scenario, result):
failures += 1
if failures:
print(f"CHAOS FAILED: {failures} scenario(s) missed SLO")
return 1 if failures else 0
if __name__ == "__main__":
import asyncio
scenarios = [
Scenario("baseline", "/search?q=documentary", "", slo_p99_ms=400),
Scenario("upstream_slow", "/search?q=documentary",
"fault=upstream_latency;rate=0.5;ms=6000", slo_p99_ms=800),
Scenario("upstream_dead", "/watch?id=abc123",
"fault=upstream_error;rate=1.0;code=503",
slo_success_rate=0.99, slo_p99_ms=500),
]
sys.exit(asyncio.run(main("https://staging.internal", scenarios)))
Notice the SLOs differ per scenario. Under upstream_slow I raise the allowed p99 to 800ms because some latency leaking through is acceptable — but success rate must stay at 99% because the circuit breaker and FTS5 fallback should absorb the failures. Under upstream_dead I actually tighten p99 to 500ms, because when the upstream is fully down the breaker should trip and fail fast to the local path. A slow response during a total outage means the breaker is not doing its job. Encoding that expectation is the whole point.
Running it against LiteSpeed and Cloudflare
We run three profiles. The first hits PHP-FPM behind LiteSpeed directly on the staging box — this isolates application behavior. The second hits the same box through Cloudflare with cache rules mirroring production, which catches the class of bug from my 3 AM story: a degraded partial response getting cached at the edge and served to thousands of users. The cache_poison fault plus the Cloudflare profile is the only reliable way I have found to reproduce that.
A few operational notes that saved us grief:
-
Never let chaos touch a shared cache key. The
X-Chaos-Specheader must be part of the cache key on staging, or one poisoned response contaminates clean requests. On LiteSpeed we add it to the cache vary list for the staging vhost only. - Warm the FTS5 page cache before the baseline run, otherwise your baseline p99 includes cold-start cost and every comparison is noise.
-
Run db-lock scenarios against a copy of the real database, not a fixture. SQLite
SQLITE_BUSYbehavior depends heavily on WAL mode and busy_timeout, and a toy DB will not reproduce it. - Keep total request counts modest (500–2000). Chaos testing is about finding behavioral bugs, not load testing; save the 100k-request runs for a separate performance suite.
We wired the harness into CI as a nightly job against staging rather than per-PR, because chaos runs take a few minutes and are inherently a little noisy. A failed nightly opens a ticket automatically with the failing scenario name and the recorded p99/success numbers attached.
What actually changed
Since this harness went in, the concrete wins were unglamorous and exactly what I wanted:
- We found that
upstream_partialresponses were passing our JSON schema check but rendering blank watch pages — the 3 AM bug, now caught in CI. - We discovered the circuit breaker was tripping correctly but the cooldown was tuned so aggressively that a single slow request kept the breaker open for real users long after the upstream recovered.
- We caught a case where a
db_lockduring cron caused search to return HTTP 200 with zero results instead of falling back to the last good cache — a silent correctness bug no unit test would ever have surfaced.
None of those were exotic. They were the ordinary seams between a fast local path and a slow remote one, exactly where distributed reads always rot. The harness did not make our system more reliable by itself — the circuit breaker and the FTS5-first design did that. What the harness gave us was proof, on demand, that the resilience we designed actually holds when things go wrong, plus a red build the moment it stops holding.
If you run any read-heavy API that stitches together caches, a local index, and flaky upstreams, start small: one fault type, one endpoint, one SLO assertion in CI. The first bug you catch during business hours instead of at 3 AM pays for the whole thing.
Top comments (0)