Building a Video CDN Edge Cache Layer With Varnish and ESI Fragments

The problem: eight regions, one melting origin

A streaming-discovery page looks like a caching dream until you actually profile it. On TrendVidStream a single catalog page is maybe 80% identical for every visitor in a region — the same trending rail, the same category chips, the same footer, the same SEO block. But the remaining 20% is poison for a naive full-page cache: the "popular in your region right now" strip rotates every few minutes as our multi-region cron ingests fresh signals, and the per-visitor "continue watching" row is unique to a session cookie.

We run eight regions. Full-page caching the whole document means every region-specific rotation invalidates a huge object, and any personalization forces Cache-Control: private, which drops the hit rate through the floor. Turning caching off entirely melts a PHP 8.4 + SQLite origin the moment a page gets shared on social.

The answer is not "cache harder." It is to stop treating the page as one object. Edge Side Includes (ESI) let you cache the page skeleton for hours while caching the volatile fragments for seconds, and let Varnish assemble the final HTML at the edge before it ever touches your origin. This post is the exact VCL, PHP, and tooling we run in production.

Why ESI beats a single full-page cache

ESI is a tiny markup dialect. Your origin emits a placeholder tag:

<esi:include src="/_esi/popular?region=DE" />

Varnish sees that tag while it's assembling the response, fetches the fragment as if it were a separate request, caches that fragment under its own key and TTL, and splices the result into the parent document. The parent and the child are independent cache objects. That single property fixes the whole problem:

• The page skeleton can be cached for 6 hours because it no longer contains anything that changes.
• The regional popular strip gets a 90-second TTL and a region-scoped key, so a rotation in Germany never invalidates the skeleton or the France fragment.
• The personalized row is marked uncacheable (or cached per-session) without contaminating anything else.
• A cache purge becomes surgical. When our FTP deploy ships a new template, we purge skeletons. When the cron ingests fresh trending data, we purge only the popular fragments.

The math is what sells it. Before ESI, our effective hit rate on catalog pages hovered around 40% because personalization forced misses. After splitting into fragments, the skeleton hits ~98% and only the small popular fragment carries real origin load — and that fragment is a 4 KB JSON-to-HTML render, not a full page build.

The Varnish VCL that makes it work

Here is the core of our default.vcl (Varnish 7.x). It enables ESI processing, decides what is cacheable, and — critically — builds a cache key that includes the region so eight regions don't stomp on each other.

vcl 4.1;

import std;

backend origin {
    .host = "127.0.0.1";
    .port = "8080";
    .connect_timeout = 2s;
    .first_byte_timeout = 10s;
}

sub vcl_recv {
    # Normalize the region into a header we control, from a Cloudflare geo
    # hint or an explicit ?region= override. Never trust the raw query alone.
    if (req.url ~ "[?&]region=") {
        set req.http.X-Region = regsub(req.url, ".*[?&]region=([A-Z]{2}).*", "\1");
    } else if (req.http.CF-IPCountry) {
        set req.http.X-Region = req.http.CF-IPCountry;
    } else {
        set req.http.X-Region = "US";
    }

    # ESI fragment requests are internal-only. Never let the public hit them
    # directly, but allow Varnish's own restart-driven subrequests through.
    if (req.url ~ "^/_esi/" && req.http.X-Varnish-ESI != "1") {
        # Public request to a fragment path -> pretend it doesn't exist.
        return (synth(404, "Not found"));
    }

    # Strip cookies for cacheable GETs so a session cookie can't force a miss
    # on the skeleton. The personalized fragment handles auth separately.
    if (req.method == "GET" && req.url !~ "^/_esi/session") {
        unset req.http.Cookie;
    }

    return (hash);
}

sub vcl_hash {
    hash_data(req.url);
    hash_data(req.http.host);
    # Region participates in the key so DE and FR are distinct objects.
    hash_data(req.http.X-Region);
}

sub vcl_backend_response {
    # Turn on ESI parsing only when the backend opts in via a header.
    if (beresp.http.X-ESI == "on") {
        set beresp.do_esi = true;
    }

    # Skeleton pages: long TTL, allow serving stale while we refetch.
    if (bereq.url !~ "^/_esi/") {
        set beresp.ttl = 6h;
        set beresp.grace = 12h;
    }

    # Fragments carry their own Cache-Control from the origin; respect it,
    # but clamp the popular strip so a bad header can't cache it forever.
    if (bereq.url ~ "^/_esi/popular") {
        set beresp.ttl = 90s;
        set beresp.grace = 30s;
    }

    if (bereq.url ~ "^/_esi/session") {
        set beresp.uncacheable = true;
        set beresp.ttl = 0s;
    }

    return (deliver);
}

sub vcl_deliver {
    set resp.http.X-Cache = obj.hits > 0 ? "HIT" : "MISS";
    set resp.http.X-Region = req.http.X-Region;
    # Don't leak internals to the browser.
    unset resp.http.X-ESI;
}

Two details earn their keep here. First, the X-Varnish-ESI gate in vcl_recv means the fragment endpoints are unreachable from the public internet — only Varnish's own ESI subrequests carry that header, which we inject on the backend side. Second, beresp.grace lets Varnish serve a slightly stale skeleton while it refetches in the background, so an origin hiccup during a cron run never produces a visible stall.

Rendering ESI fragments from PHP 8.4

The origin needs two things: emit the parent document with <esi:include> tags and the X-ESI: on header, and serve each fragment path as its own tiny endpoint. With PHP 8.4 the fragment renderer is clean — typed properties, readonly, and a match expression for TTL policy.

<?php
declare(strict_types=1);

// public/_esi.php — front controller for /_esi/* fragment routes.

final readonly class FragmentPolicy
{
    public function __construct(
        public int $ttl,
        public bool $private,
    ) {}
}

function policyFor(string $name): FragmentPolicy
{
    return match ($name) {
        'popular' => new FragmentPolicy(ttl: 90,  private: false),
        'chips'   => new FragmentPolicy(ttl: 21600, private: false), // 6h
        'session' => new FragmentPolicy(ttl: 0,   private: true),
        default   => new FragmentPolicy(ttl: 300, private: false),
    };
}

$name   = preg_replace('/[^a-z]/', '', $_GET['f'] ?? '');
$region = preg_match('/^[A-Z]{2}$/', $_GET['region'] ?? '')
    ? $_GET['region']
    : 'US';

$policy = policyFor($name);

header('Content-Type: text/html; charset=utf-8');
if ($policy->private) {
    header('Cache-Control: private, no-store');
} else {
    header("Cache-Control: public, max-age={$policy->ttl}");
}

// Render the requested fragment. Each renderer returns a small HTML string.
echo match ($name) {
    'popular' => renderPopular($region),
    'chips'   => renderChips($region),
    'session' => renderSession($_COOKIE['sid'] ?? null),
    default   => http_response_code(404),
};

function renderPopular(string $region): string
{
    $db = new PDO('sqlite:/var/www/data/catalog.db');
    $db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);

    // SQLite FTS5-backed ranking table, filtered to the caller's region.
    $stmt = $db->prepare(<<<SQL
        SELECT title, slug, thumb
        FROM popular_by_region
        WHERE region = :region
        ORDER BY score DESC
        LIMIT 12
    SQL);
    $stmt->execute([':region' => $region]);

    $out = '<div class="vw-popular" data-region="' . htmlspecialchars($region) . '">';
    foreach ($stmt as $row) {
        $out .= sprintf(
            '<a class="vw-card" href="/watch/%s"><img loading="lazy" src="%s" alt="%s"></a>',
            rawurlencode($row['slug']),
            htmlspecialchars($row['thumb']),
            htmlspecialchars($row['title']),
        );
    }
    return $out . '</div>';
}

And the parent template just drops placeholders where the volatile content used to be inline. Note the X-ESI: on header — that is the single switch that tells Varnish to parse this document for esi: tags:

<?php
declare(strict_types=1);

// templates/category.php (excerpt)
header('X-ESI: on');
$region = $_SERVER['HTTP_X_REGION'] ?? 'US';
?>
<main class="vw-catalog">
  <section class="vw-chips">
    <esi:include src="/_esi/chips?f=chips&amp;region=<?= htmlspecialchars($region) ?>" />
  </section>

  <h1>Trending on TrendVidStream</h1>
  <section class="vw-popular-wrap">
    <esi:include src="/_esi/popular?f=popular&amp;region=<?= htmlspecialchars($region) ?>" />
  </section>

  <!-- Personalized, never cached as part of the page. -->
  <aside class="vw-continue">
    <esi:include src="/_esi/session?f=session" />
  </aside>
</main>

The skeleton around these tags — the header, hero, footer, structured-data JSON-LD — is now completely static per region, which is why it can sit in Varnish for six hours. The only thing that expires quickly is the 4 KB popular fragment.

Getting the region into the key without cache explosion

The subtle trap with multi-region ESI is cardinality. If you naively Vary on a raw geo header, you can end up with 240 country variants of every object and a cache that never warms. We deliberately collapse to the eight regions we actually serve, and we do it in vcl_recv before hashing so the key carries the region rather than a Vary header. A short PHP-side normalizer mirrors the same eight-region map the cron uses, so a request tagged NL and one tagged BE both resolve to the EU-West bucket and share one cached fragment:

<?php
declare(strict_types=1);

function canonicalRegion(string $country): string
{
    static $map = [
        'US' => 'US', 'CA' => 'US',
        'GB' => 'UK', 'IE' => 'UK',
        'DE' => 'EU', 'FR' => 'EU', 'NL' => 'EU', 'BE' => 'EU',
        'JP' => 'APAC', 'KR' => 'APAC', 'SG' => 'APAC',
        'BR' => 'LATAM', 'AU' => 'OCE',
    ];
    return $map[$country] ?? 'US';
}

Eight logical regions times a handful of fragment types is a cache that fits comfortably in RAM and reaches steady state in minutes, not days.

Warming the cache from the multi-region cron

An empty cache after a deploy means the first visitor in each region eats a cold origin render. We already run a multi-region ingestion cron; warming is a natural extra step. This Python worker hits each fragment through Varnish with the internal ESI header so the objects are populated before real traffic arrives.

#!/usr/bin/env python3
"""warm_edge.py — pre-populate Varnish fragments after a cron ingest or deploy."""
import concurrent.futures as cf
import sys
import urllib.request

VARNISH = "http://127.0.0.1:6081"
REGIONS = ["US", "UK", "EU", "APAC", "LATAM", "OCE"]
FRAGMENTS = [
    ("/_esi/popular", "popular"),
    ("/_esi/chips", "chips"),
]


def warm(path: str, frag: str, region: str) -> tuple[str, int]:
    url = f"{VARNISH}{path}?f={frag}&region={region}"
    req = urllib.request.Request(url, headers={
        # This header is what unlocks the internal-only fragment route.
        "X-Varnish-ESI": "1",
        "User-Agent": "edge-warmer/1.0",
    })
    with urllib.request.urlopen(req, timeout=8) as resp:
        return (f"{region}:{frag}", resp.status)


def main() -> int:
    jobs = [
        (path, frag, region)
        for path, frag in FRAGMENTS
        for region in REGIONS
    ]
    failures = 0
    with cf.ThreadPoolExecutor(max_workers=8) as pool:
        futures = [pool.submit(warm, *job) for job in jobs]
        for fut in cf.as_completed(futures):
            try:
                label, status = fut.result()
                mark = "ok" if status == 200 else f"HTTP {status}"
                print(f"  warmed {label:16s} {mark}")
                failures += status != 200
            except Exception as exc:  # noqa: BLE001
                print(f"  warm failed: {exc}")
                failures += 1
    print(f"done, {failures} failure(s)")
    return 1 if failures else 0


if __name__ == "__main__":
    sys.exit(main())

We wire this into the same cron that refreshes popular_by_region, immediately after the SQLite write commits. By the time the ranking data changes, the fragments are re-warmed with the new data and the 90-second TTL means the old objects age out cleanly.

Surgical purging on FTP deploy

Our deploy path is FTP-based, which means there is no fancy orchestration hook — files land, and then a small step fires a purge. The important discipline is to purge only what changed. A template change invalidates skeletons; it must not blow away the fragment cache and cause a thundering-herd of origin renders across eight regions at once.

Varnish supports a PURGE method (wired via an acl in VCL, omitted above for space) and a ban for pattern invalidation. This Go tool issues targeted bans so a deploy purges skeletons while leaving warm fragments intact:

package main

import (
    "fmt"
    "net/http"
    "os"
    "time"
)

const varnishAdmin = "http://127.0.0.1:6081"

// ban issues a Varnish ban against objects whose URL matches the expression.
func ban(expr string) error {
    req, err := http.NewRequest("BAN", varnishAdmin+"/", nil)
    if err != nil {
        return err
    }
    req.Header.Set("X-Ban-Expr", expr)

    client := &http.Client{Timeout: 5 * time.Second}
    resp, err := client.Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    if resp.StatusCode != http.StatusOK {
        return fmt.Errorf("ban %q returned %s", expr, resp.Status)
    }
    return nil
}

func main() {
    scope := "skeleton"
    if len(os.Args) > 1 {
        scope = os.Args[1]
    }

    // Map a deploy scope to the narrowest ban we can get away with.
    var expr string
    switch scope {
    case "skeleton":
        // Everything except fragment routes: keep warm fragments alive.
        expr = `req.url !~ "^/_esi/"`
    case "fragments":
        expr = `req.url ~ "^/_esi/"`
    case "all":
        expr = `req.url ~ ".*"`
    default:
        fmt.Fprintf(os.Stderr, "unknown scope %q\n", scope)
        os.Exit(2)
    }

    if err := ban(expr); err != nil {
        fmt.Fprintf(os.Stderr, "purge failed: %v\n", err)
        os.Exit(1)
    }
    fmt.Printf("purged scope=%s expr=%q\n", scope, expr)
}

The deploy script calls ./purge skeleton after uploading templates and ./purge fragments only when a fragment renderer changed. Because bans are lazy — Varnish evaluates the ban expression against an object the next time it's looked up — there is no synchronous stall, and warm fragments simply survive the deploy.

What we measured

Rolling out ESI fragmentation on catalog pages changed the numbers in ways that mattered:

• Origin request volume on catalog routes dropped roughly 12x. The skeleton stopped being rebuilt per visitor; only the small popular fragment carried live load, and even that hit ~85% at its 90-second TTL.
• p95 time-to-first-byte from the edge went from ~180 ms (origin render) to under 15 ms for skeleton hits, because Varnish assembles the ESI document from RAM.
• Deploys stopped causing latency spikes. Scoped bans plus fragment warming meant a template ship no longer triggered a cold-origin stampede across eight regions.
• Personalization got cheaper, not more expensive. The session fragment is the only uncacheable piece, so it's a 2 KB render instead of forcing the entire page private.

There are real costs to be honest about. ESI adds a moving part: fragment endpoints are a second surface to secure (hence the internal-only header gate), and debugging an assembled page means checking which of several cache objects went stale. You also give up on caching truly personalized fragments — those still hit origin every time, so keep them small and rare. And ESI is not free CPU: Varnish parses the parent document on every miss to find the include tags, so keep the tag count modest.

Conclusion

The mental shift that makes edge caching work for a multi-region video-discovery site is to stop asking "can I cache this page" and start asking "which parts of this page change together." Once you decompose along those seams, ESI lets each part live at its own TTL under its own key: a six-hour skeleton, a ninety-second regional strip, an uncacheable personalized row — all stitched at the edge in single-digit milliseconds.

The stack around it stays boring on purpose. PHP 8.4 renders small typed fragment endpoints, SQLite FTS5 answers the ranking queries, the multi-region cron warms the cache the moment it writes new data, and an FTP deploy fires a scoped Varnish ban instead of nuking everything. None of those pieces are exotic, and that is the point: the cleverness lives in the cache-key design and the fragment boundaries, not in the infrastructure. Start by carving one volatile strip out of your hottest page, give it its own TTL, and watch your origin load fall off a cliff.