Building a Video CDN Edge Cache Layer with Varnish and ESI Fragments

#varnish #caching #php #performance

Our watch pages looked static, so we cached them hard at the edge. Then support tickets started: logged-in users saw "Sign in" in the header, watch counts froze at whatever number they had when the page was first cached, and the "Recommended for you" rail showed the same twelve videos to everyone on the planet. The page was 95% identical for every visitor and 5% deadly personal, and a single Cache-Control: public, max-age=3600 treated all of it the same. That 5% is what makes edge caching a video-platform problem instead of a blog-caching problem. On DailyWatch we solved it with Varnish and Edge Side Includes (ESI), and this post is the actual VCL, the PHP fragment endpoints, and the invalidation plumbing we run in production.

The short version: stop thinking of a page as one cacheable object. Think of it as a cacheable skeleton with a few holes, and let the edge assemble the holes at request time. That is exactly what ESI does — and Varnish is one of the few caches that implements it well.

Why Cloudflare Alone Was Not Enough

Our stack is PHP 8.4 behind LiteSpeed, SQLite with FTS5 for search, and Cloudflare in front. Cloudflare's edge cache is excellent for truly static assets — thumbnails, CSS, the video poster JPEGs — but it caches whole responses keyed by URL. It has no first-class way to say "cache this HTML for an hour, but re-render this one <div> per user." You can approximate it with Cloudflare Workers and subrequests, but you end up writing an ESI engine by hand.

Varnish already is that engine. So we run a Varnish tier between Cloudflare and our origin: Cloudflare handles TLS, DDoS, and static assets; Varnish handles HTML assembly and micro-caching; LiteSpeed/PHP renders fragments. The request path looks like this:

Cloudflare edge (static assets, TLS termination, no HTML caching)
Varnish (HTML skeleton cache + ESI assembly + invalidation)
LiteSpeed + PHP 8.4 (renders skeleton and each fragment)
SQLite (data)

The key mental shift: the skeleton and each fragment are separate cache objects in Varnish, each with its own TTL and its own invalidation key. A watch page skeleton can live for six hours while its view-count fragment lives for thirty seconds, and neither one forces the other to re-render.

Splitting a Watch Page Into Fragments

Before any VCL, decide the seams. On a DailyWatch watch page, the fragments are:

Skeleton — title, description, embedded player, editorial text. Changes only when an editor edits the video. TTL: 6h.
Header — either "Sign in" or the user's avatar and menu. Per-user. TTL: private, short.
View count / stats — updates constantly. TTL: 30s, shared across all users.
Recommendation rail — personalized when logged in, popularity-based when anonymous. TTL: 5m anonymous, private when logged in.

The skeleton is plain HTML with <esi:include> tags where the dynamic holes go. Here is the PHP that renders it. Note there is zero user-specific data in this output — that is the whole point, it has to be cacheable by everyone.

<?php
declare(strict_types=1);

// watch.php — renders the CACHEABLE skeleton only.
// No session_start(), no per-user branches. Anonymous-safe by construction.

function render_watch_skeleton(array $video): string
{
    $id    = htmlspecialchars($video['id'], ENT_QUOTES);
    $title = htmlspecialchars($video['title'], ENT_QUOTES);
    $desc  = htmlspecialchars($video['description'], ENT_QUOTES);

    // esi:include points at internal fragment endpoints.
    // Varnish fetches + caches each one independently.
    return <<<HTML
    <!doctype html>
    <html lang="en">
    <head><title>{$title} — DailyWatch</title></head>
    <body>
      <header>
        <esi:include src="/_frag/header" />
      </header>

      <main>
        <h1>{$title}</h1>
        <div class="player" data-video="{$id}"></div>
        <p class="desc">{$desc}</p>

        <div class="stats">
          <esi:include src="/_frag/stats/{$id}" />
        </div>

        <aside class="recs">
          <esi:include src="/_frag/recs/{$id}" />
        </aside>
      </main>
    </body>
    </html>
    HTML;
}

The skeleton response must tell Varnish two things: that ESI processing is required, and how long the skeleton itself may be cached. We do that with a Surrogate-Control header and a normal Cache-Control:

<?php
// Emitted alongside the skeleton HTML above.
header('Surrogate-Control: content="ESI/1.0"');   // "please process my esi tags"
header('Cache-Control: public, max-age=21600');    // skeleton good for 6h
header('X-Fragment: skeleton');                     // used for invalidation keys

Surrogate-Control: content="ESI/1.0" is the standard, spec-defined way for an origin to opt a response into ESI parsing. Without it, Varnish will not scan the body for <esi:include> tags, which is a good safety default — you never want to accidentally interpret user-generated content as ESI markup.

The VCL That Ties It Together

Here is the core of our default.vcl (Varnish 7.x). It does four jobs: enable ESI when the origin asks for it, key fragments correctly, make the personalized fragments bypass the shared cache, and expose a PURGE/BAN interface for invalidation.

vcl 4.1;

backend origin {
    .host = "127.0.0.1";
    .port = "8088";        # LiteSpeed listening locally
}

# Only our origin and localhost may purge.
acl purgers {
    "127.0.0.1";
    "10.0.0.0"/8;
}

sub vcl_recv {
    # Invalidation entrypoints -------------------------------------
    if (req.method == "PURGE") {
        if (client.ip !~ purgers) { return (synth(405, "Not allowed")); }
        return (purge);
    }
    if (req.method == "BAN") {
        if (client.ip !~ purgers) { return (synth(405, "Not allowed")); }
        # Ban every object whose X-Video-Id matches the header we send.
        ban("obj.http.X-Video-Id == " + req.http.X-Video-Id);
        return (synth(200, "Banned"));
    }

    # Personalized header fragment must never be shared -------------
    if (req.url == "/_frag/header" || req.url ~ "^/_frag/recs/") {
        if (req.http.Cookie ~ "sid=") {
            # Logged-in: go straight to origin, do not cache.
            return (pass);
        }
    }

    # Strip cookies for everything we intend to cache anonymously.
    if (req.url ~ "^/_frag/" || req.url ~ "^/watch/") {
        unset req.http.Cookie;
    }

    return (hash);
}

sub vcl_backend_response {
    # Turn on ESI parsing only when the origin explicitly asked.
    if (beresp.http.Surrogate-Control ~ "ESI/1.0") {
        set beresp.do_esi = true;
        unset beresp.http.Surrogate-Control;
    }

    # Per-fragment TTLs, driven by an origin header.
    if (bereq.url ~ "^/_frag/stats/")  { set beresp.ttl = 30s; }
    if (bereq.url ~ "^/_frag/recs/")   { set beresp.ttl = 300s; }
    if (bereq.url ~ "^/watch/")        { set beresp.ttl = 6h; }

    # Short grace: serve slightly stale while we refetch in the
    # background. Keeps p99 flat during origin hiccups.
    set beresp.grace = 60s;
    return (deliver);
}

sub vcl_deliver {
    set resp.http.X-Cache = obj.hits > 0 ? "HIT" : "MISS";
}

A few things worth calling out, because they are the parts people get wrong:

ESI is opt-in per response. beresp.do_esi is only set when we see the Surrogate-Control header. This means our fragment endpoints, which return plain HTML with no ESI tags, are never scanned — a tiny but real CPU saving at scale, and a security boundary.

Personalized fragments pass, they do not cache. When a request carries a session cookie and hits /_frag/header or /_frag/recs/, we return (pass), which sends it to origin every time and stores nothing. The skeleton is still a shared cache HIT; only the two personal holes go to origin. So a logged-in user's watch page is one origin request for the header plus one for recs, versus a full page render. That is the entire economic argument for ESI.

Grace mode is not optional for video. beresp.grace = 60s lets Varnish serve a just-expired object while it revalidates in the background. Without it, every TTL expiry becomes a synchronous origin hit, and under a traffic spike those pile up into a thundering herd. Grace turns a cliff into a ramp.

The Stats Fragment: Cheap, Hot, and Shared

The view-count fragment is requested on every watch page load, so it must be trivially cheap to render and safe to share across all users. It reads one integer from SQLite and returns a snippet:

<?php
declare(strict_types=1);

// _frag/stats/{id} — shared, 30s TTL. No cookies, no session.
$id = $_GET['id'] ?? '';
if (!preg_match('/^[A-Za-z0-9_-]{6,20}$/', $id)) {
    http_response_code(400);
    exit;
}

$db = new PDO('sqlite:/var/www/data/videos.db');
$db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);

$stmt = $db->prepare('SELECT views FROM videos WHERE id = :id');
$stmt->execute([':id' => $id]);
$views = (int) ($stmt->fetchColumn() ?: 0);

// Tell Varnish exactly how to cache and how to ban this object.
header('Cache-Control: public, max-age=30');
header('X-Video-Id: ' . $id);   // enables targeted BAN on edit
header('Content-Type: text/html; charset=utf-8');

printf('<span class="views">%s views</span>', number_format($views));

Notice the X-Video-Id header. Varnish stores response headers with the cached object, and our BAN logic in the VCL matches on obj.http.X-Video-Id. That means when an editor updates a video, we can invalidate every fragment and skeleton tied to that video with a single ban — the stats, the recs, and the skeleton all carry the same X-Video-Id, so one command clears them all without knowing their URLs.

Invalidation Without Tears

Caching is easy; invalidation is where teams lose weeks. Our rule: content changes push, they do not wait for TTL. When the CMS saves a video, or the cron job that ingests new videos runs, it sends a BAN to Varnish. Here is the small Go daemon we use — it sits next to the app, receives an internal event, and fires the ban. Go because it is a single static binary we can drop on the Varnish host with no runtime.

package main

import (
    "fmt"
    "log"
    "net/http"
    "time"
)

// banVideo tells Varnish to drop every cached object tagged with this
// video id: skeleton, stats fragment, and recs fragment in one shot.
func banVideo(varnishAddr, videoID string) error {
    req, err := http.NewRequest("BAN", "http://"+varnishAddr+"/", nil)
    if err != nil {
        return err
    }
    req.Header.Set("X-Video-Id", videoID)

    client := &http.Client{Timeout: 3 * time.Second}
    resp, err := client.Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK {
        return fmt.Errorf("ban failed: %s", resp.Status)
    }
    return nil
}

func main() {
    // Internal endpoint the PHP app calls after a save/ingest.
    http.HandleFunc("/invalidate", func(w http.ResponseWriter, r *http.Request) {
        id := r.URL.Query().Get("video")
        if id == "" {
            http.Error(w, "missing video", http.StatusBadRequest)
            return
        }
        if err := banVideo("127.0.0.1:6081", id); err != nil {
            log.Printf("ban %s: %v", id, err)
            http.Error(w, "ban failed", http.StatusBadGateway)
            return
        }
        log.Printf("invalidated video %s", id)
        w.WriteHeader(http.StatusNoContent)
    })

    log.Fatal(http.ListenAndServe("127.0.0.1:9099", nil))
}

The PHP side calls this after any write. Because we ban by X-Video-Id rather than by URL, we never have to enumerate the URLs of the fragments — a single logical event maps to a single ban, no matter how many cache objects it touches. This is the biggest operational win of the tag-based approach over URL purging: your invalidation logic stops caring about your URL structure.

One caution on bans: Varnish evaluates the ban list lazily, checking each cached object against outstanding bans on its next lookup. If you fire thousands of bans per minute the ban list grows and slows lookups. For a video platform where edits are rare relative to reads, this is a non-issue — we ban a few hundred times a day. If you were banning constantly, you would switch to PURGE on known URLs or use the ban lurker to keep the list trimmed.

Measuring Whether It Actually Works

We added X-Cache: HIT/MISS in vcl_deliver so we can measure the hit ratio per fragment type, and varnishstat gives the aggregate. The numbers that matter for us:

Skeleton hit ratio: ~99%. It only misses right after an edit or a cold start. This is the bulk of the HTML bytes, so it dominates origin-load savings.
Stats fragment hit ratio: ~94%. With a 30s TTL, on a page doing thousands of views a minute, almost every request is a HIT and the origin sees roughly two SQLite reads per minute per video instead of one per pageview.
Anonymous recs hit ratio: ~90%. Five-minute TTL, popularity-based, shared.
Logged-in fragments: 0% by design. They pass, and that is correct — but they are small and fast, so the origin cost is bounded.

The headline result: origin PHP requests for watch pages dropped by roughly 20x, because one cached skeleton now serves what used to be a full render, and only the tiny personal holes reach PHP. SQLite, which is single-writer and can get contended, mostly sees the 30-second stats read now instead of a full query set per pageview.

Things That Bit Us

Cookies leaking into the cache key. Early on, an analytics cookie was part of the hash, so the anonymous skeleton was cached per-visitor — hit ratio near zero. Stripping cookies in vcl_recv for cacheable URLs fixed it instantly. Always audit what is actually in your cache key with varnishlog -g request.

ESI includes failing silently. If a fragment endpoint returns a 500, the default ESI behavior is to insert nothing, so the page renders with a blank hole and no error. We now set onerror="continue" explicitly and monitor fragment 5xx rates separately, because a page that looks 95% fine hides the broken 5% from every dashboard that only watches top-level status codes.

Double caching with Cloudflare. We had to make sure Cloudflare does not cache the HTML — otherwise it caches the fully-assembled page including one user's header. Our watch HTML goes out Cache-Control: public for Varnish but we use a Cloudflare cache rule to bypass HTML, letting Cloudflare cache only static assets. Getting the two tiers to agree on who owns HTML caching is the subtle part of running both.

Conclusion

ESI is old technology — the spec dates to 2001 — but it solves a genuinely modern problem: pages that are almost entirely shared with a few genuinely personal pixels. Instead of choosing between "cache the whole page and lie to users" or "cache nothing and melt the origin," you cache the 95% aggressively and let the edge stitch in the 5% at request time. For a free video discovery platform where read traffic dwarfs writes, that split cut our origin load by an order of magnitude while keeping view counts fresh and headers correct per user.

Start small: pick your most-viewed page, find its seams, split out the two or three genuinely dynamic holes, and give each one its own TTL. Add tag-based invalidation with a header like X-Video-Id so a single content event clears everything related to it. Measure hit ratio per fragment, not per page. The VCL above is close to what we run in production, and it has held up through traffic spikes that would have taken the origin down in the pre-ESI days.