DEV Community: ahmet gedik

WebSocket Real-Time Notifications for Video Platforms with Python

ahmet gedik — Sat, 02 May 2026 15:00:01 +0000

Real-Time in a Multi-Region Video Platform

When a viral news clip appears in the UAE or a Danish documentary breaks into Nordic trending charts, TrendVidStream users should see it instantly. A Python WebSocket server bridges the gap between the PHP cron fetcher and the browser.

Architecture

PHP Cron Fetcher ──→ Redis Pub/Sub ──→ Python WS Server ──→ Browser clients
     (8 regions)        (one channel per region)           (filtered by locale)

Python WebSocket Server

pip install websockets redis asyncio

import asyncio
import json
import logging
import urllib.parse
import redis.asyncio as aioredis
import websockets
from websockets.server import WebSocketServerProtocol

logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO)

region_clients: dict[str, set[WebSocketServerProtocol]] = {}

async def broadcast_to_region(region: str, message: str) -> None:
    clients = region_clients.get(region, set())
    if not clients:
        return
    dead = set()
    results = await asyncio.gather(
        *(ws.send(message) for ws in clients), return_exceptions=True,
    )
    for ws, result in zip(list(clients), results):
        if isinstance(result, Exception):
            dead.add(ws)
    clients.difference_update(dead)

async def handler(websocket: WebSocketServerProtocol) -> None:
    qs = urllib.parse.parse_qs(urllib.parse.urlparse(websocket.path).query)
    region = qs.get('region', ['US'])[0].upper()
    region_clients.setdefault(region, set()).add(websocket)
    try:
        await websocket.wait_closed()
    finally:
        region_clients.get(region, set()).discard(websocket)

async def redis_subscriber() -> None:
    client = aioredis.from_url('redis://localhost:6379', decode_responses=True)
    pubsub = client.pubsub()
    channels = ['tvs:AE', 'tvs:FI', 'tvs:CZ', 'tvs:DK', 'tvs:BE', 'tvs:GB', 'tvs:CH', 'tvs:US']
    await pubsub.subscribe(*channels)

    async for msg in pubsub.listen():
        if msg['type'] != 'message':
            continue
        region = msg['channel'].split(':')[1]
        try:
            payload = json.loads(msg['data'])
            out = json.dumps(payload, ensure_ascii=False)
            await broadcast_to_region(region, out)
        except (json.JSONDecodeError, KeyError) as e:
            logger.warning(f'Bad payload: {e}')

async def main() -> None:
    server = await websockets.serve(handler, '0.0.0.0', 8765)
    await asyncio.gather(server.wait_closed(), redis_subscriber())

if __name__ == '__main__':
    asyncio.run(main())

PHP Publisher with RTL Metadata

<?php
declare(strict_types=1);

class NotificationPublisher
{
    private \Redis $redis;

    public function __construct()
    {
        $this->redis = new \Redis();
        $this->redis->connect('127.0.0.1', 6379);
    }

    public function publish(array $video, string $region): void
    {
        $payload = json_encode([
            'event'     => 'new_trending',
            'region'    => $region,
            'video_id'  => $video['video_id'],
            'title'     => $video['title'],
            'thumbnail' => $video['thumbnail_url'],
            'channel'   => $video['channel_title'],
            'rtl'       => in_array($region, ['AE', 'IL'], true),
            'ts'        => time(),
        ], JSON_UNESCAPED_UNICODE);

        $this->redis->publish("tvs:{$region}", $payload);
    }
}

Browser Client with RTL Toast

class TrendingNotifier {
  constructor(region = 'AE') {
    this.ws = new WebSocket(`wss://ws.trendvidstream.com/?region=${region}`);
    this.ws.onmessage = (e) => {
      const data = JSON.parse(e.data);
      if (data.event === 'new_trending') this.showToast(data);
    };
  }

  showToast(video) {
    const toast = document.createElement('div');
    toast.className = 'trending-toast';
    if (video.rtl) toast.setAttribute('dir', 'rtl');
    toast.innerHTML = `<img src="${video.thumbnail}" alt=""><div><strong>Trending in ${video.region}</strong><span>${video.title}</span></div>`;
    document.body.appendChild(toast);
    setTimeout(() => toast.remove(), 5000);
  }
}

new TrendingNotifier(currentUserRegion);

Multi-Region Considerations

TrendVidStream spans UAE (Arabic RTL), Czech Republic (diacritics: č, ž, š), Finland (ä, ö), and Denmark (ø, å). Key points:

Arabic RTL: send rtl: true in the payload so the browser can apply dir="rtl" dynamically.
JSON encoding: always JSON_UNESCAPED_UNICODE in PHP and ensure_ascii=False in Python so diacritics and Arabic glyphs transmit unmangled.
Connection per region: routing clients into per-region sets means a viral clip in Finland only wakes Finnish clients, not UAE users.

This article is part of the Building TrendVidStream series. Check out TrendVidStream to see these techniques in action.

WebSocket Real-Time Notifications for Video Platforms with Python

ahmet gedik — Sat, 02 May 2026 08:00:01 +0000

The Case for Real-Time Viral Alerts

When a video is going viral across European regions, every minute counts. At ViralVidVault, our cron pipeline detects virality spikes across Poland, Netherlands, Sweden, Norway, Austria, and the UK -- but without real-time push, users only see updates on their next page refresh. WebSockets fix that.

Architecture Overview

The data flows through three stages:

Cron detects a virality spike and sends an HTTP POST to the WebSocket server
WebSocket server broadcasts a viral alert to all connected browsers
Clients render a real-time notification with video details

The WebSocket Server

import asyncio
import json
import websockets
from datetime import datetime
from typing import Set, Dict

CONNECTIONS: Set[websockets.WebSocketServerProtocol] = set()
REGION_SUBS: Dict[str, Set[websockets.WebSocketServerProtocol]] = {}

async def handler(websocket: websockets.WebSocketServerProtocol):
    """Handle client connections with optional region subscription."""
    CONNECTIONS.add(websocket)
    subscribed_regions = set()

    try:
        async for message in websocket:
            data = json.loads(message)

            if data.get("type") == "subscribe_region":
                region = data["region"].upper()
                subscribed_regions.add(region)
                REGION_SUBS.setdefault(region, set()).add(websocket)
                await websocket.send(json.dumps({
                    "type": "subscribed",
                    "region": region,
                }))

            elif data.get("type") == "ping":
                await websocket.send(json.dumps({"type": "pong"}))

    except websockets.ConnectionClosed:
        pass
    finally:
        CONNECTIONS.discard(websocket)
        for region in subscribed_regions:
            REGION_SUBS.get(region, set()).discard(websocket)

async def broadcast_viral(video: dict, region: str):
    """Send viral alert to clients subscribed to the region or all."""
    message = json.dumps({
        "type": "viral_alert",
        "region": region,
        "video": video,
        "timestamp": datetime.now().isoformat(),
    })

    # Send to region subscribers
    targets = REGION_SUBS.get(region, set()) | REGION_SUBS.get("ALL", set())

    # Fallback: if no subscribers, broadcast to everyone
    if not targets:
        targets = CONNECTIONS.copy()

    await asyncio.gather(
        *(ws.send(message) for ws in targets),
        return_exceptions=True,
    )

async def http_webhook(reader: asyncio.StreamReader, writer: asyncio.StreamWriter):
    """Accept webhook POST from cron pipeline."""
    data = await reader.read(65536)
    body = data.split(b"\r\n\r\n", 1)[-1]

    try:
        payload = json.loads(body)
        await broadcast_viral(
            video=payload["video"],
            region=payload.get("region", "EU"),
        )
        writer.write(b"HTTP/1.1 200 OK\r\n\r\nOK")
    except (json.JSONDecodeError, KeyError):
        writer.write(b"HTTP/1.1 400 Bad Request\r\n\r\nBad JSON")

    await writer.drain()
    writer.close()

async def main():
    ws = await websockets.serve(handler, "0.0.0.0", 8765)
    http = await asyncio.start_server(http_webhook, "0.0.0.0", 8766)
    print(f"WS on :8765, webhook on :8766")
    await asyncio.gather(ws.wait_closed(), http.serve_forever())

if __name__ == "__main__":
    asyncio.run(main())

The key enhancement here over a basic broadcast is region-based subscriptions. Users browsing Polish content only get alerts about Polish viral videos.

Client-Side Implementation

class ViralAlertClient {
  constructor(wsUrl, regions = []) {
    this.wsUrl = wsUrl;
    this.regions = regions;
    this.reconnectMs = 1000;
    this.connect();
  }

  connect() {
    this.ws = new WebSocket(this.wsUrl);

    this.ws.onopen = () => {
      this.reconnectMs = 1000;
      // Subscribe to specific regions
      this.regions.forEach(region => {
        this.ws.send(JSON.stringify({ type: 'subscribe_region', region }));
      });
      this.heartbeat = setInterval(() => {
        this.ws.send(JSON.stringify({ type: 'ping' }));
      }, 25000);
    };

    this.ws.onmessage = (event) => {
      const data = JSON.parse(event.data);
      if (data.type === 'viral_alert') {
        this.showViralToast(data);
      }
    };

    this.ws.onclose = () => {
      clearInterval(this.heartbeat);
      setTimeout(() => this.connect(), this.reconnectMs);
      this.reconnectMs = Math.min(this.reconnectMs * 2, 30000);
    };
  }

  showViralToast(data) {
    const { video, region } = data;
    const toast = document.createElement('div');
    toast.className = 'viral-toast';
    toast.innerHTML = `
      <img src="${video.thumbnail}" alt="" width="80">
      <div>
        <strong>Viral in ${region}!</strong>
        <p>${video.title}</p>
      </div>
    `;
    document.body.appendChild(toast);
    setTimeout(() => toast.classList.add('show'), 10);
    setTimeout(() => { toast.classList.remove('show'); setTimeout(() => toast.remove(), 300); }, 6000);
  }
}

// Subscribe to European regions
const alerts = new ViralAlertClient('wss://viralvidvault.com/ws', ['PL', 'NL', 'SE', 'NO', 'AT', 'GB']);

Triggering from the Virality Scorer

Integrate with the existing ViralVidVault scoring pipeline:

import requests

def on_viral_detected(video: dict, region: str, score: float):
    if score < 85.0:
        return  # Only alert on truly viral content

    payload = {
        "region": region,
        "video": {
            "id": video["video_id"],
            "title": video["title"],
            "thumbnail": video["thumbnail_url"],
            "score": score,
        },
    }
    try:
        requests.post("http://localhost:8766", json=payload, timeout=3)
    except requests.RequestException:
        pass  # Non-critical, don't fail the pipeline

Region-aware WebSocket notifications bring the real-time experience that makes ViralVidVault feel alive. Users browsing Swedish content get Swedish viral alerts instantly, without polling.

This article is part of the Building ViralVidVault series. Check out ViralVidVault to see these techniques in action.

WebSocket Real-Time Notifications for Video Platforms with Python

ahmet gedik — Fri, 01 May 2026 20:00:01 +0000

Why Real-Time Notifications Matter for Video Platforms

When a K-pop music video suddenly goes viral in South Korea, or a Japanese gaming stream breaks into the trending charts, users of TopVideoHub should know about it within seconds — not the next time they refresh. WebSockets make that possible.

This article walks through building a WebSocket notification server that broadcasts "new trending video" events to connected clients as soon as the cron fetcher discovers them.

Architecture Overview

Cron Fetcher (PHP) ──→ Redis Pub/Sub ──→ Python WS Server ──→ Browser clients

The PHP cron fetcher publishes events to Redis. A lightweight Python WebSocket server subscribes to Redis and fans those events out to all connected browsers. This decoupling means the fetcher never blocks waiting for browser connections.

Python WebSocket Server

Install dependencies:

pip install websockets redis asyncio

The server:

import asyncio
import json
import logging
import redis.asyncio as aioredis
import websockets
from websockets.server import WebSocketServerProtocol

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

connected: set[WebSocketServerProtocol] = set()

async def broadcast(message: str) -> None:
    if not connected:
        return
    dead = set()
    results = await asyncio.gather(
        *(ws.send(message) for ws in connected),
        return_exceptions=True,
    )
    for ws, result in zip(list(connected), results):
        if isinstance(result, Exception):
            dead.add(ws)
    connected.difference_update(dead)

async def handler(websocket: WebSocketServerProtocol) -> None:
    connected.add(websocket)
    region = websocket.request_headers.get('X-Region', 'US')
    logger.info(f'Client connected (region={region}), total={len(connected)}')
    try:
        await websocket.wait_closed()
    finally:
        connected.discard(websocket)

async def redis_subscriber() -> None:
    client = aioredis.from_url('redis://localhost:6379', decode_responses=True)
    pubsub = client.pubsub()
    await pubsub.subscribe('topvideohub:trending')

    async for message in pubsub.listen():
        if message['type'] != 'message':
            continue
        try:
            payload = json.loads(message['data'])
            payload['title'] = payload.get('title', '')
            await broadcast(json.dumps(payload, ensure_ascii=False))
        except (json.JSONDecodeError, KeyError) as e:
            logger.warning(f'Bad message: {e}')

async def main() -> None:
    server = await websockets.serve(handler, '0.0.0.0', 8765)
    logger.info('WebSocket server listening on :8765')
    await asyncio.gather(server.wait_closed(), redis_subscriber())

if __name__ == '__main__':
    asyncio.run(main())

Publishing Events from PHP

After inserting a new trending video, the PHP fetcher publishes to Redis:

<?php
declare(strict_types=1);

class NotificationPublisher
{
    private \Redis $redis;

    public function __construct()
    {
        $this->redis = new \Redis();
        $this->redis->connect('127.0.0.1', 6379);
    }

    public function publishTrending(array $video, string $region): void
    {
        $payload = json_encode([
            'event'     => 'new_trending',
            'region'    => $region,
            'video_id'  => $video['video_id'],
            'title'     => $video['title'],
            'thumbnail' => $video['thumbnail_url'],
            'channel'   => $video['channel_title'],
            'ts'        => time(),
        ], JSON_UNESCAPED_UNICODE);

        $this->redis->publish('topvideohub:trending', $payload);
    }
}

// In the fetch loop:
$publisher = new NotificationPublisher();
foreach ($newVideos as $video) {
    $db->insertVideo($video, $region);
    $publisher->publishTrending($video, $region);
}

Browser Client

class TrendingNotifier {
  constructor(region = 'JP') {
    this.region = region;
    this.ws = null;
    this.reconnectDelay = 1000;
    this.connect();
  }

  connect() {
    this.ws = new WebSocket('wss://ws.topvideohub.com/');
    this.ws.onopen = () => { this.reconnectDelay = 1000; };
    this.ws.onmessage = (event) => {
      const data = JSON.parse(event.data);
      if (data.event === 'new_trending') this.showToast(data);
    };
    this.ws.onclose = () => {
      setTimeout(() => this.connect(), this.reconnectDelay);
      this.reconnectDelay = Math.min(this.reconnectDelay * 2, 30000);
    };
  }

  showToast(video) {
    const toast = document.createElement('div');
    toast.className = 'trending-toast';
    toast.innerHTML = `<img src="${video.thumbnail}" alt=""><div><strong>New in ${video.region}</strong><span>${video.title}</span></div>`;
    document.body.appendChild(toast);
    setTimeout(() => toast.remove(), 5000);
  }
}

const notifier = new TrendingNotifier(userRegion);

CJK Considerations

Asian video titles contain multi-byte characters. Two pitfalls to avoid:

PHP: always use JSON_UNESCAPED_UNICODE so "title": "新しい動画" stays readable.
Python: pass ensure_ascii=False to json.dumps() for the same reason.
WebSocket frame size: CJK strings are longer in bytes (3 bytes/char UTF-8). Keep notification payloads small.

Connection Lifecycle and Scaling

For TopVideoHub, which serves 9 regions, a single Python process handles hundreds of concurrent WebSocket connections comfortably. If you need to scale horizontally, replace the in-process connected set with a Redis-backed pub/sub fan-out.

This article is part of the Building TopVideoHub series. Check out TopVideoHub to see these techniques in action.

Rate Limiting and API Key Management for Video Data APIs

ahmet gedik — Fri, 01 May 2026 11:00:01 +0000

The YouTube Data API v3 gives you 10,000 quota units per day per key. That sounds generous until you realize a single search request costs 100 units and a video details call costs 1 unit per video. Do the math for a platform that fetches trending content from 8 regions every 2 hours, and you hit the wall fast. Here's how DailyWatch handles it.

Understanding YouTube API Quotas

Not all API calls are equal. The quota cost varies by endpoint:

Endpoint	Cost per call
`search.list`	100 units
`videos.list`	1 unit
`channels.list`	1 unit
`playlistItems.list`	1 unit

A single search.list call returning 50 results costs the same 100 units as one returning 5. Always request maxResults=50 to maximize value per call.

Multi-Key Rotation

One key gives 10,000 units. Three keys give 30,000. The rotation logic is straightforward:

class ApiKeyManager {
    private array $keys;
    private PDO $db;

    public function __construct(PDO $db) {
        $this->db = $db;
        $this->keys = $this->loadKeys();
    }

    private function loadKeys(): array {
        $stmt = $this->db->query(
            "SELECT api_key, daily_usage, last_reset 
             FROM api_keys 
             WHERE active = 1 
             ORDER BY daily_usage ASC"
        );
        return $stmt->fetchAll(PDO::FETCH_ASSOC);
    }

    public function getNextKey(): ?string {
        foreach ($this->keys as $key) {
            // Reset counter if new day (Pacific Time, matching Google's reset)
            if ($this->isNewDay($key['last_reset'])) {
                $this->resetUsage($key['api_key']);
                return $key['api_key'];
            }
            // Skip exhausted keys
            if ($key['daily_usage'] < 9500) { // 500 unit buffer
                return $key['api_key'];
            }
        }
        return null; // All keys exhausted
    }

    public function recordUsage(string $apiKey, int $units): void {
        $stmt = $this->db->prepare(
            "UPDATE api_keys 
             SET daily_usage = daily_usage + ?, last_used = datetime('now') 
             WHERE api_key = ?"
        );
        $stmt->execute([$units, $apiKey]);
    }
}

Keys are sorted by usage ascending, so the least-used key always gets picked first. The 500-unit buffer prevents accidentally exceeding the quota on the last call of the day.

Rate Limiting the Fetch Process

Even with multiple keys, you don't want to fire 50 API calls in a tight loop. Google enforces per-second rate limits too:

class RateLimiter {
    private float $lastCallTime = 0;
    private float $minInterval;

    public function __construct(float $callsPerSecond = 5.0) {
        $this->minInterval = 1.0 / $callsPerSecond;
    }

    public function wait(): void {
        $elapsed = microtime(true) - $this->lastCallTime;
        if ($elapsed < $this->minInterval) {
            usleep((int)(($this->minInterval - $elapsed) * 1_000_000));
        }
        $this->lastCallTime = microtime(true);
    }
}

// Usage in fetch loop
$limiter = new RateLimiter(3.0); // 3 calls per second
$keyManager = new ApiKeyManager($db);

foreach ($regions as $region) {
    $limiter->wait();
    $apiKey = $keyManager->getNextKey();
    if (!$apiKey) {
        log_message('All API keys exhausted, stopping fetch');
        break;
    }
    $results = fetchTrending($apiKey, $region);
    $keyManager->recordUsage($apiKey, 100); // search cost
}

Quota-Efficient Fetching

The biggest optimization is reducing search calls. Instead of searching per category per region, batch smartly:

// BAD: 8 regions x 15 categories = 120 search calls = 12,000 units
foreach ($regions as $region) {
    foreach ($categories as $category) {
        searchVideos($region, $category); // 100 units each
    }
}

// GOOD: Use chart=mostPopular (1 unit) + videoCategoryId filter
foreach ($regions as $region) {
    // 1 unit instead of 100!
    $popular = getPopularVideos($apiKey, $region, 50);
    foreach ($categories as $category) {
        $catVideos = getPopularVideos($apiKey, $region, 50, $category);
    }
}

The videos.list endpoint with chart=mostPopular costs 1 unit versus 100 for search.list. For 8 regions with 15 categories, that's 128 units instead of 12,000.

Handling 403 Quota Exceeded

When a key hits its limit, Google returns HTTP 403 with a specific error reason. Handle it gracefully:

function makeApiCall(ApiKeyManager $km, string $url): ?array {
    $maxRetries = count($km->getKeys());
    for ($i = 0; $i < $maxRetries; $i++) {
        $key = $km->getNextKey();
        if (!$key) return null;

        $response = httpGet($url . '&key=' . $key);

        if ($response['status'] === 200) {
            return json_decode($response['body'], true);
        }

        if ($response['status'] === 403) {
            $error = json_decode($response['body'], true);
            $reason = $error['error']['errors'][0]['reason'] ?? '';

            if ($reason === 'quotaExceeded' || $reason === 'dailyLimitExceeded') {
                $km->markExhausted($key);
                continue; // Try next key
            }
        }
    }
    return null;
}

The loop automatically rotates to the next available key on quota errors. If all keys are exhausted, it returns null and the caller decides whether to retry later.

Monitoring and Alerts

Track daily usage in the database and alert when approaching limits:

function checkQuotaHealth(PDO $db): array {
    $stmt = $db->query(
        "SELECT api_key, daily_usage,
                ROUND(daily_usage * 100.0 / 10000, 1) AS usage_pct
         FROM api_keys WHERE active = 1"
    );
    $keys = $stmt->fetchAll(PDO::FETCH_ASSOC);
    $totalUsed = array_sum(array_column($keys, 'daily_usage'));
    $totalAvailable = count($keys) * 10000;

    return [
        'keys_active'    => count($keys),
        'total_used'     => $totalUsed,
        'total_available' => $totalAvailable,
        'usage_percent'  => round($totalUsed / $totalAvailable * 100, 1),
        'per_key'        => $keys,
    ];
}

On DailyWatch, this data feeds into a simple admin dashboard. When combined usage crosses 80%, the system automatically reduces fetch frequency for lower-priority regions.

The key insight: treat API quotas as a finite resource to be budgeted, not a limit to be worked around. Design your fetch strategy around the quota, not the other way around.

This article is part of the Building DailyWatch series.

Rate Limiting and API Key Management for YouTube Data API

ahmet gedik — Fri, 01 May 2026 02:00:01 +0000

TrendVidStream covers 8 regions across three continents: US, GB, CH, DK, AE, BE, CZ, FI. The YouTube Data API v3 has a hard 10,000-unit daily quota per API key. With 8 regions to refresh and user-initiated searches to serve, quota management is not optional.

Here is the complete system.

Quota Cost Reference

Operation	Cost	Notes
`videos.list` (trending)	1 unit	Called per region per cron
`search.list`	100 units	Expensive — always cache results
`videoCategories.list`	1 unit	Cached 24h
`videos.list` (by ID)	1 unit	Watch page stale refresh

With 8 regions, a cron every 7 hours, and 3 cron cycles per day: 8 regions × 3 cycles = 24 units/day for trending. The budget is dominated by search.

Database Schema

CREATE TABLE api_keys (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    key_val     TEXT NOT NULL UNIQUE,
    provider    TEXT NOT NULL DEFAULT 'youtube',
    quota_used  INTEGER NOT NULL DEFAULT 0,
    quota_limit INTEGER NOT NULL DEFAULT 10000,
    quota_date  TEXT DEFAULT (DATE('now')),  -- Track daily reset
    is_active   INTEGER NOT NULL DEFAULT 1,
    error_count INTEGER NOT NULL DEFAULT 0,
    last_error  TEXT,
    last_used   TEXT
);

CREATE TABLE quota_log (
    id         INTEGER PRIMARY KEY AUTOINCREMENT,
    key_id     INTEGER NOT NULL REFERENCES api_keys(id),
    operation  TEXT NOT NULL,
    region     TEXT,
    units      INTEGER NOT NULL DEFAULT 1,
    logged_at  TEXT DEFAULT (datetime('now'))
);

CREATE INDEX idx_quota_log_date ON quota_log(logged_at);
CREATE INDEX idx_quota_log_region ON quota_log(region, logged_at);

Key Manager

<?php

class ApiKeyManager
{
    private PDO $db;
    private ?array $current = null;

    public function __construct(PDO $db)
    {
        $this->db = $db;
        $this->resetExpiredQuotas();
    }

    private function resetExpiredQuotas(): void
    {
        // YouTube quota resets at midnight Pacific Time
        // We use the date stored on the row to detect day rollover
        $this->db->exec(<<<SQL
            UPDATE api_keys
            SET quota_used  = 0,
                error_count = 0,
                is_active   = 1,
                quota_date  = DATE('now')
            WHERE quota_date < DATE('now')
              AND provider = 'youtube'
        SQL);
    }

    public function getBestKey(int $unitsNeeded = 1): ?array
    {
        if ($this->current !== null &&
            ($this->current['quota_limit'] - $this->current['quota_used']) >= $unitsNeeded) {
            return $this->current;
        }

        $stmt = $this->db->prepare(<<<SQL
            SELECT id, key_val, quota_used, quota_limit,
                   (quota_limit - quota_used) AS remaining
            FROM api_keys
            WHERE provider  = 'youtube'
              AND is_active  = 1
              AND (quota_limit - quota_used) >= ?
            ORDER BY remaining DESC
            LIMIT 1
        SQL);
        $stmt->execute([$unitsNeeded]);

        $this->current = $stmt->fetch(PDO::FETCH_ASSOC) ?: null;
        return $this->current;
    }

    public function consume(int $keyId, string $operation, ?string $region, int $units): void
    {
        $this->db->prepare(<<<SQL
            UPDATE api_keys
            SET quota_used = quota_used + ?,
                last_used  = datetime('now')
            WHERE id = ?
        SQL)->execute([$units, $keyId]);

        $this->db->prepare(<<<SQL
            INSERT INTO quota_log (key_id, operation, region, units)
            VALUES (?, ?, ?, ?)
        SQL)->execute([$keyId, $operation, $region, $units]);

        $this->current = null; // Invalidate cached key
    }

    public function markError(int $keyId, string $error): void
    {
        $this->db->prepare(<<<SQL
            UPDATE api_keys
            SET error_count = error_count + 1,
                last_error  = ?,
                is_active   = CASE WHEN error_count + 1 >= 3 THEN 0 ELSE is_active END
            WHERE id = ?
        SQL)->execute([$error, $keyId]);

        $this->current = null;
    }

    public function getUsageSummary(): array
    {
        return $this->db->query(<<<SQL
            SELECT
                k.key_val,
                k.quota_used,
                k.quota_limit,
                ROUND(100.0 * k.quota_used / k.quota_limit, 1) AS pct_used,
                k.is_active,
                (
                    SELECT SUM(l.units)
                    FROM quota_log l
                    WHERE l.key_id = k.id
                      AND DATE(l.logged_at) = DATE('now')
                      AND l.operation = 'search.list'
                ) AS search_units_today
            FROM api_keys k
            WHERE k.provider = 'youtube'
            ORDER BY k.quota_used DESC
        SQL)->fetchAll(PDO::FETCH_ASSOC);
    }
}

YouTube Client with Fallback

<?php

class YouTubeClient
{
    private ApiKeyManager $keys;
    private const MAX_RETRIES = 3;

    public function fetchTrending(string $region, int $maxResults = 50): array
    {
        return $this->callApi('videos.list', [
            'part'       => 'snippet,statistics',
            'chart'      => 'mostPopular',
            'regionCode' => $region,
            'maxResults' => $maxResults,
        ], cost: 1, region: $region);
    }

    public function search(string $query, string $region): array
    {
        return $this->callApi('search', [
            'part'       => 'snippet',
            'q'          => $query,
            'regionCode' => $region,
            'type'       => 'video',
            'maxResults' => 20,
        ], cost: 100, region: $region);
    }

    private function callApi(
        string $endpoint,
        array  $params,
        int    $cost,
        string $region
    ): array {
        $key = $this->keys->getBestKey($cost);
        if ($key === null) {
            // No quota — return graceful empty instead of crashing
            error_log("[TVS] All API keys exhausted. Returning cached/empty for {$region}");
            return [];
        }

        $params['key'] = $key['key_val'];
        $url = "https://www.googleapis.com/youtube/v3/{$endpoint}?" . http_build_query($params);

        $ch = curl_init($url);
        curl_setopt_array($ch, [
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_TIMEOUT        => 15,
            CURLOPT_HTTPHEADER     => ['Accept: application/json'],
        ]);
        $body = curl_exec($ch);
        $code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
        curl_close($ch);

        if ($code === 403) {
            $err = json_decode($body, true);
            $reason = $err['error']['errors'][0]['reason'] ?? 'unknown';

            if ($reason === 'quotaExceeded') {
                // Force-exhaust this key so it rotates away
                $this->keys->consume($key['id'], $endpoint, $region,
                    $key['quota_limit'] - $key['quota_used']);
                // Retry with next available key
                return $this->callApi($endpoint, $params, $cost, $region);
            }

            $this->keys->markError($key['id'], "403:{$reason}");
            return [];
        }

        if ($code !== 200) {
            $this->keys->markError($key['id'], "HTTP:{$code}");
            return [];
        }

        $this->keys->consume($key['id'], $endpoint, $region, $cost);
        return json_decode($body, true)['items'] ?? [];
    }
}

Per-Region Search Budget

AE (UAE) and GB tend to get more search traffic. Protect the quota:

<?php

const REGION_SEARCH_BUDGET = [
    'AE' => 3000,  // UAE — high Middle East traffic
    'GB' => 2000,  // UK — strong European traffic
    'US' => 2000,
    'CH' => 800,
    'DK' => 600,
    'BE' => 600,
    'CZ' => 400,
    'FI' => 400,
];

function canSearch(string $region, PDO $db): bool
{
    $budget = REGION_SEARCH_BUDGET[$region] ?? 200;

    $used = $db->prepare(<<<SQL
        SELECT COALESCE(SUM(units), 0)
        FROM quota_log
        WHERE region = ?
          AND operation = 'search'
          AND DATE(logged_at) = DATE('now')
    SQL);
    $used->execute([$region]);
    $usedToday = (int)$used->fetchColumn();

    return ($usedToday + 100) <= $budget;
}

Admin Quota Dashboard

<?php
// GET /ibt — Admin panel quota section

$summary = $keyManager->getUsageSummary();
foreach ($summary as $key) {
    printf(
        "Key: ...%s | Used: %d/%d (%s%%) | Active: %s | Search today: %d units\n",
        substr($key['key_val'], -8),
        $key['quota_used'],
        $key['quota_limit'],
        $key['pct_used'],
        $key['is_active'] ? 'yes' : 'NO',
        $key['search_units_today'] ?? 0
    );
}

Auto-Import from Config

; api_keys.conf — deployed to all sites by ops.sh
youtube_key=AIzaSyABC123...
youtube_key=AIzaSyDEF456...
youtube_key=AIzaSyGHI789...

<?php

public function autoImportApiKeys(): void
{
    $conf = @parse_ini_file(__DIR__ . '/../api_keys.conf', false, INI_SCANNER_RAW);
    if (!$conf) return;

    $stmt = $this->pdo->prepare(<<<SQL
        INSERT OR IGNORE INTO api_keys (key_val, provider)
        VALUES (?, 'youtube')
    SQL);

    foreach ((array)($conf['youtube_key'] ?? []) as $key) {
        $stmt->execute([trim($key)]);
    }
}

With 3 keys and a combined 30,000 daily units, TrendVidStream refreshes all 8 regions on schedule while serving hundreds of cached search queries per day — well within budget.

This is part of the "Building TrendVidStream" series, documenting the architecture behind a global video directory covering Nordic, Middle Eastern, and Central European regions.

Rate Limiting and API Key Management for Video Data APIs

ahmet gedik — Thu, 30 Apr 2026 16:00:01 +0000

YouTube's Data API v3 quota is 10,000 units per key per day. For ViralVidVault, which fetches trending videos across seven European regions every seven hours, that quota needs careful management. One careless loop can burn through an entire day's allocation in minutes.

The Quota Math

Understand what each call costs before writing any fetch logic:

API Call	Quota Cost
`search.list`	100 units
`videos.list`	1 unit
`channels.list`	1 unit
`playlistItems.list`	1 unit

A naive approach — searching each category in each region — means 7 regions times 15 categories times 100 units = 10,500 units per fetch cycle. That's more than one key allows per day.

Key Rotation Architecture

Multiple API keys, rotated by usage. Store usage data in the same SQLite database that holds video data:

class YouTubeKeyManager {
    private PDO $db;

    public function __construct(PDO $db) {
        $this->db = $db;
        $this->resetExpiredKeys();
    }

    private function resetExpiredKeys(): void {
        // Google resets quotas at midnight Pacific Time
        $this->db->exec(
            "UPDATE api_keys 
             SET daily_usage = 0, last_reset = datetime('now')
             WHERE last_reset < datetime('now', '-1 day')"
        );
    }

    public function getBestKey(): ?array {
        $stmt = $this->db->query(
            "SELECT id, api_key, daily_usage 
             FROM api_keys 
             WHERE active = 1 AND daily_usage < 9500
             ORDER BY daily_usage ASC 
             LIMIT 1"
        );
        return $stmt->fetch(PDO::FETCH_ASSOC) ?: null;
    }

    public function recordUsage(int $keyId, int $units): void {
        $stmt = $this->db->prepare(
            "UPDATE api_keys 
             SET daily_usage = daily_usage + ?,
                 total_usage = total_usage + ?,
                 last_used = datetime('now')
             WHERE id = ?"
        );
        $stmt->execute([$units, $units, $keyId]);
    }
}

The getBestKey() method selects the key with the lowest daily usage, spreading the load evenly. The 9,500-unit threshold leaves a 500-unit buffer to prevent accidental overruns.

Rate Limiter Class

Google enforces per-second limits in addition to daily quotas. A simple token bucket keeps you under the wire:

class TokenBucketLimiter {
    private float $tokens;
    private float $maxTokens;
    private float $refillRate;
    private float $lastRefill;

    public function __construct(float $requestsPerSecond = 3.0) {
        $this->maxTokens = $requestsPerSecond;
        $this->tokens = $requestsPerSecond;
        $this->refillRate = $requestsPerSecond;
        $this->lastRefill = microtime(true);
    }

    public function acquire(): void {
        $this->refill();
        while ($this->tokens < 1.0) {
            usleep(50000); // 50ms
            $this->refill();
        }
        $this->tokens -= 1.0;
    }

    private function refill(): void {
        $now = microtime(true);
        $elapsed = $now - $this->lastRefill;
        $this->tokens = min($this->maxTokens, $this->tokens + $elapsed * $this->refillRate);
        $this->lastRefill = $now;
    }
}

Unlike a simple sleep(1) between calls, the token bucket allows bursts up to the limit and then throttles smoothly.

Quota-Efficient Fetch Strategy

The biggest win is avoiding search.list entirely when possible. The videos.list endpoint with chart=mostPopular costs just 1 unit:

function fetchRegionContent(
    YouTubeKeyManager $km,
    TokenBucketLimiter $limiter,
    string $region
): array {
    $results = [];

    // 1 unit per call instead of 100
    $key = $km->getBestKey();
    $limiter->acquire();

    $popular = youtubeGet('videos', [
        'part'       => 'snippet,statistics',
        'chart'      => 'mostPopular',
        'regionCode' => $region,
        'maxResults' => 50,
        'key'        => $key['api_key'],
    ]);
    $km->recordUsage($key['id'], 1);
    $results = array_merge($results, $popular['items'] ?? []);

    // Per-category popular (also 1 unit each)
    $categories = ['1', '2', '10', '17', '20', '22', '23', '24', '25', '26', '28'];
    foreach ($categories as $catId) {
        $key = $km->getBestKey();
        if (!$key) break;

        $limiter->acquire();
        $catVideos = youtubeGet('videos', [
            'part'            => 'snippet,statistics',
            'chart'           => 'mostPopular',
            'regionCode'      => $region,
            'videoCategoryId' => $catId,
            'maxResults'      => 50,
            'key'             => $key['api_key'],
        ]);
        $km->recordUsage($key['id'], 1);
        $results = array_merge($results, $catVideos['items'] ?? []);
    }

    return $results;
}

For 7 regions with 11 categories each, that's 7 * 12 = 84 units per fetch cycle. Compare that to 10,500 units with the search.list approach. A 125x improvement.

Handling Quota Errors Gracefully

When a key hits its limit, YouTube returns a 403 with a specific error reason. Catch it and rotate:

function youtubeGetWithRetry(
    YouTubeKeyManager $km,
    TokenBucketLimiter $limiter,
    string $endpoint,
    array $params
): ?array {
    $attempts = 0;
    $maxAttempts = 3;

    while ($attempts < $maxAttempts) {
        $key = $km->getBestKey();
        if (!$key) {
            error_log('All YouTube API keys exhausted');
            return null;
        }

        $params['key'] = $key['api_key'];
        $limiter->acquire();
        $response = httpGet(buildYouTubeUrl($endpoint, $params));

        if ($response['status'] === 200) {
            return json_decode($response['body'], true);
        }

        if ($response['status'] === 403) {
            $body = json_decode($response['body'], true);
            $reason = $body['error']['errors'][0]['reason'] ?? 'unknown';
            if (in_array($reason, ['quotaExceeded', 'dailyLimitExceeded'])) {
                $km->recordUsage($key['id'], 10000); // Mark as exhausted
                $attempts++;
                continue;
            }
        }

        error_log("YouTube API error: HTTP {$response['status']}");
        return null;
    }
    return null;
}

Monitoring Dashboard

Track quota consumption to catch issues before they become outages:

function getQuotaDashboard(PDO $db): array {
    $keys = $db->query(
        "SELECT api_key, daily_usage, last_used, last_reset,
                ROUND(daily_usage * 100.0 / 10000, 1) AS pct
         FROM api_keys WHERE active = 1
         ORDER BY daily_usage DESC"
    )->fetchAll(PDO::FETCH_ASSOC);

    $totalUsed = array_sum(array_column($keys, 'daily_usage'));
    $totalCap = count($keys) * 10000;

    return [
        'total_used'     => $totalUsed,
        'total_capacity' => $totalCap,
        'pct_used'       => round($totalUsed / $totalCap * 100, 1),
        'keys'           => $keys,
    ];
}

On ViralVidVault, the admin panel displays this in real time. When aggregate usage crosses 70%, the system logs a warning. At 90%, it reduces fetch frequency to preserve quota for the rest of the day.

Key Takeaway

The YouTube API quota is not a problem to solve once — it's a constraint to design around from the start. Choose cheap endpoints over expensive ones, rotate keys evenly, and monitor consumption. The difference between a platform that runs smoothly and one that goes dark at 3pm because it burned through its quota is just a few hundred lines of management code.

This article is part of the Building ViralVidVault series.

Rate Limiting and API Key Management for YouTube Data API

ahmet gedik — Thu, 30 Apr 2026 08:00:01 +0000

TopVideoHub fetches trending video data from 9 Asia-Pacific regions — US, GB, JP, KR, TW, SG, VN, TH, HK. Each YouTube Data API v3 videos.list call costs 1 unit. Each search.list call costs 100 units. With a default quota of 10,000 units per key per day, we need careful management to keep all 9 regions refreshed.

Here is the full key rotation and rate limiting system.

Quota Anatomy

Operation	Units	Usage
`videos.list` (trending, 50 results)	1	Every cron cycle per region
`search.list` (keyword)	100	On-demand, cached 6h
`videoCategories.list`	1	Daily, cached 24h

With 9 regions and a 4-hour cron cycle, trending fetches cost 9 × 6 = 54 units/day. Negligible. The budget killer is search — 100 units per unique query.

Key Storage in SQLite

CREATE TABLE api_keys (
    id       INTEGER PRIMARY KEY AUTOINCREMENT,
    key_val  TEXT NOT NULL UNIQUE,
    provider TEXT NOT NULL DEFAULT 'youtube',
    quota_used   INTEGER NOT NULL DEFAULT 0,
    quota_limit  INTEGER NOT NULL DEFAULT 10000,
    quota_reset  TEXT,           -- ISO 8601 UTC timestamp of next reset
    is_active    INTEGER NOT NULL DEFAULT 1,
    error_count  INTEGER NOT NULL DEFAULT 0,
    last_error   TEXT,
    created_at   TEXT DEFAULT (datetime('now'))
);

CREATE INDEX idx_api_keys_active ON api_keys(provider, is_active, quota_used);

QuotaManager

<?php

class QuotaManager
{
    private PDO $db;
    private ?array $activeKey = null;

    public function __construct(PDO $db)
    {
        $this->db = $db;
    }

    /**
     * Get the key with most remaining quota.
     * Returns null if all keys are exhausted.
     */
    public function getActiveKey(): ?array
    {
        if ($this->activeKey !== null) {
            return $this->activeKey;
        }

        $stmt = $this->db->prepare(<<<SQL
            SELECT *
            FROM api_keys
            WHERE provider = 'youtube'
              AND is_active = 1
              AND (quota_used < quota_limit)
              AND (
                  quota_reset IS NULL
                  OR quota_reset < datetime('now')
                  OR quota_used < quota_limit
              )
            ORDER BY (quota_limit - quota_used) DESC
            LIMIT 1
        SQL);
        $stmt->execute();

        $this->activeKey = $stmt->fetch(PDO::FETCH_ASSOC) ?: null;
        return $this->activeKey;
    }

    public function consume(string $keyVal, int $units): void
    {
        $this->db->prepare(<<<SQL
            UPDATE api_keys
            SET quota_used = quota_used + ?,
                quota_reset = COALESCE(
                    quota_reset,
                    datetime('now', '+1 day', 'start of day')
                )
            WHERE key_val = ?
        SQL)->execute([$units, $keyVal]);

        // Bust the in-memory cache so next call re-fetches
        $this->activeKey = null;
    }

    public function markError(string $keyVal, string $error): void
    {
        $this->db->prepare(<<<SQL
            UPDATE api_keys
            SET error_count = error_count + 1,
                last_error  = ?,
                is_active   = CASE WHEN error_count >= 5 THEN 0 ELSE is_active END
            WHERE key_val = ?
        SQL)->execute([$error, $keyVal]);

        $this->activeKey = null;
    }

    public function canFetch(int $unitsNeeded = 1): bool
    {
        $key = $this->getActiveKey();
        if ($key === null) {
            return false;
        }
        return ($key['quota_limit'] - $key['quota_used']) >= $unitsNeeded;
    }

    public function resetDailyQuotas(): void
    {
        // Called at midnight UTC by cron
        $this->db->exec(<<<SQL
            UPDATE api_keys
            SET quota_used = 0,
                error_count = 0,
                is_active = 1,
                quota_reset = NULL
            WHERE quota_reset < datetime('now')
        SQL);
    }
}

YouTube API Client with Rotation

<?php

class YouTubeApi
{
    private QuotaManager $quota;
    private int $retryDelay = 2; // seconds

    public function __construct(QuotaManager $quota)
    {
        $this->quota = $quota;
    }

    public function fetchTrending(string $region, int $maxResults = 50): array
    {
        $key = $this->quota->getActiveKey();
        if ($key === null) {
            throw new QuotaExhaustedException("All API keys exhausted for region {$region}");
        }

        $url = 'https://www.googleapis.com/youtube/v3/videos?' . http_build_query([
            'part'           => 'snippet,statistics,contentDetails',
            'chart'          => 'mostPopular',
            'regionCode'     => $region,
            'maxResults'     => $maxResults,
            'videoCategoryId' => '',
            'key'            => $key['key_val'],
        ]);

        $response = $this->get($url);

        if ($response['http_code'] === 403) {
            // Quota exceeded — disable this key, try next
            $this->quota->markError($key['key_val'], 'quota_exceeded');
            $this->quota->consume($key['key_val'], $key['quota_limit']); // Force exhaustion
            return $this->fetchTrending($region, $maxResults); // Recurse with next key
        }

        if ($response['http_code'] !== 200) {
            $this->quota->markError($key['key_val'], "HTTP {$response['http_code']}");
            throw new ApiException("YouTube API error: {$response['http_code']}");
        }

        // Charge the quota: 1 unit for videos.list
        $this->quota->consume($key['key_val'], 1);

        $body = json_decode($response['body'], true);
        return $body['items'] ?? [];
    }

    private function get(string $url): array
    {
        $ch = curl_init($url);
        curl_setopt_array($ch, [
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_TIMEOUT        => 20,
            CURLOPT_FOLLOWLOCATION => true,
            CURLOPT_HTTPHEADER     => ['Accept: application/json'],
        ]);
        $body     = curl_exec($ch);
        $httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
        $error    = curl_error($ch);
        curl_close($ch);

        if ($error) {
            throw new NetworkException("cURL error: {$error}");
        }

        return ['http_code' => $httpCode, 'body' => $body];
    }
}

Per-Region Budget Enforcement

Search queries are expensive (100 units each). Budget them per region:

<?php

class RegionBudget
{
    // Units per day per region for search
    private const BUDGETS = [
        'JP' => 2000, // High-traffic
        'KR' => 2000,
        'US' => 1500,
        'GB' => 1000,
        'TW' => 800,
        'SG' => 600,
        'HK' => 600,
        'VN' => 400,
        'TH' => 400,
    ];

    public function canSearch(string $region): bool
    {
        $used = $this->getUsedToday($region);
        $budget = self::BUDGETS[$region] ?? 200;
        return $used + 100 <= $budget;
    }

    private function getUsedToday(string $region): int
    {
        // Read from SQLite daily_budget table
        $stmt = $this->db->prepare(<<<SQL
            SELECT COALESCE(SUM(units_used), 0)
            FROM quota_log
            WHERE region = ? AND DATE(used_at) = DATE('now')
        SQL);
        $stmt->execute([$region]);
        return (int) $stmt->fetchColumn();
    }
}

Circuit Breaker

<?php

class CircuitBreaker
{
    private const THRESHOLD   = 5;   // failures before opening
    private const TIMEOUT     = 300; // seconds before half-open

    public function isOpen(string $service): bool
    {
        $state = $this->getState($service);
        if ($state['status'] === 'open') {
            if (time() - $state['opened_at'] > self::TIMEOUT) {
                $this->halfOpen($service);
                return false; // Allow one test request
            }
            return true; // Still open
        }
        return false;
    }

    public function recordSuccess(string $service): void
    {
        $this->setState($service, ['status' => 'closed', 'failures' => 0]);
    }

    public function recordFailure(string $service): void
    {
        $state = $this->getState($service);
        $failures = ($state['failures'] ?? 0) + 1;

        if ($failures >= self::THRESHOLD) {
            $this->setState($service, [
                'status'    => 'open',
                'failures'  => $failures,
                'opened_at' => time(),
            ]);
            error_log("Circuit breaker OPEN for {$service}");
        } else {
            $this->setState($service, ['status' => 'closed', 'failures' => $failures]);
        }
    }
}

Auto-Import from Config File

<?php

// api_keys.conf (deployed alongside the app)
// youtube_key=AIzaSyABC123...
// youtube_key=AIzaSyDEF456...
// youtube_key=AIzaSyGHI789...

public function autoImportApiKeys(): void
{
    $conf = parse_ini_file(__DIR__ . '/../api_keys.conf', false, INI_SCANNER_RAW);
    $keys = (array)($conf['youtube_key'] ?? []);

    $stmt = $this->pdo->prepare(<<<SQL
        INSERT OR IGNORE INTO api_keys (key_val, provider)
        VALUES (?, 'youtube')
    SQL);

    foreach ($keys as $key) {
        $stmt->execute([trim($key)]);
    }
}

With 3 YouTube API keys and a combined 30,000 daily units, TopVideoHub can refresh all 9 Asia-Pacific regions on schedule every day with budget to spare for user-initiated searches.

This is part of the "Building TopVideoHub" series, documenting the architecture behind a video discovery platform covering 9 Asia-Pacific regions.

CI/CD Pipeline for a Multi-Site Video Platform

ahmet gedik — Wed, 29 Apr 2026 21:00:01 +0000

Deploying a video platform to four separate hosting environments used to mean running a script manually and hoping nothing went wrong. Here's how I replaced that with a GitHub Actions CI/CD pipeline for DailyWatch and its sibling sites.

The Problem

Four sites, each on LiteSpeed shared hosting. Same codebase, different configurations (.env files, API keys, region settings). Deployments were FTP-based — no SSH, no Docker on the hosts, no Kubernetes. Just good old file transfer.

Manual deploys meant:

Forgetting to deploy to one site
Deploying stale code after switching branches
No rollback mechanism

GitHub Actions Workflow Structure

The workflow triggers on pushes to main and runs three stages: lint, test, deploy.

# .github/workflows/deploy.yml
name: Deploy Multi-Site

on:
  push:
    branches: [main]
  workflow_dispatch:  # Manual trigger

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: shivammathur/setup-php@v2
        with:
          php-version: '8.3'
          extensions: sqlite3, pdo_sqlite, curl
      - name: PHP Lint
        run: find app/ public/ -name '*.php' -exec php -l {} \;
      - name: Check .htaccess syntax
        run: |
          apache2ctl -t -f /dev/null -c "Include $(pwd)/public/.htaccess" 2>&1 || true

  test:
    needs: lint
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: shivammathur/setup-php@v2
        with:
          php-version: '8.3'
          extensions: sqlite3, pdo_sqlite, curl, zip
      - name: Run Tests
        run: |
          php tests/run.php
        env:
          APP_ENV: testing

  deploy:
    needs: test
    runs-on: ubuntu-latest
    strategy:
      matrix:
        site: [dwv, tvh, vvv, tvs]
      fail-fast: false  # Don't cancel other deploys if one fails
    steps:
      - uses: actions/checkout@v4
      - name: Deploy to ${{ matrix.site }}
        uses: SamKirkland/FTP-Deploy-Action@v4.3.5
        with:
          server: ${{ secrets[format('{0}_FTP_HOST', matrix.site)] }}
          username: ${{ secrets[format('{0}_FTP_USER', matrix.site)] }}
          password: ${{ secrets[format('{0}_FTP_PASS', matrix.site)] }}
          server-dir: /public_html/
          exclude: |
            **/.env
            **/data/**
            **/*.log
            **/.git/**
            **/tests/**

The matrix strategy runs four deploy jobs in parallel. fail-fast: false ensures a failure on one site doesn't cancel the others — if tvh fails, dwv, vvv, and tvs still deploy.

FTP Secrets Management

Each site gets three secrets in GitHub repository settings:

DWV_FTP_HOST, DWV_FTP_USER, DWV_FTP_PASS
TVH_FTP_HOST, TVH_FTP_USER, TVH_FTP_PASS
VVV_FTP_HOST, VVV_FTP_USER, VVV_FTP_PASS
TVS_FTP_HOST, TVS_FTP_USER, TVS_FTP_PASS

The format() function in the workflow dynamically selects the right secret based on the matrix value. No hardcoded credentials in the YAML.

Post-Deploy: Cache Clearing

LiteSpeed caches aggressively. After deploy, clear it:

      - name: Clear LiteSpeed Cache
        uses: SamKirkland/FTP-Deploy-Action@v4.3.5
        with:
          server: ${{ secrets[format('{0}_FTP_HOST', matrix.site)] }}
          username: ${{ secrets[format('{0}_FTP_USER', matrix.site)] }}
          password: ${{ secrets[format('{0}_FTP_PASS', matrix.site)] }}
          server-dir: /lscache/
          dangerous-clean-slate: true  # Wipe the directory

Alternatively, hit a cache-clear endpoint after deploy:

      - name: Trigger Cache Clear
        run: |
          SITE_URL=${{ secrets[format('{0}_URL', matrix.site)] }}
          curl -sf "${SITE_URL}/task/clear-cache?key=${{ secrets.TASK_KEY }}" || true

Deploy Verification

Deploying is only half the job. Verify the deploy actually worked:

      - name: Verify Deployment
        run: |
          SITE_URL=${{ secrets[format('{0}_URL', matrix.site)] }}
          STATUS=$(curl -s -o /dev/null -w '%{http_code}' "${SITE_URL}")
          if [ "$STATUS" != "200" ]; then
            echo "::error::Site returned HTTP $STATUS after deploy"
            exit 1
          fi
          # Check version marker
          VERSION=$(curl -sf "${SITE_URL}/health" | jq -r '.version')
          echo "Deployed version: $VERSION"

The health endpoint returns the current git commit hash, so you can confirm the deploy propagated.

Notifications

Add a Slack or Discord notification at the end:

  notify:
    needs: deploy
    runs-on: ubuntu-latest
    if: always()
    steps:
      - name: Send Notification
        run: |
          STATUS="success"
          if [ "${{ needs.deploy.result }}" != "success" ]; then
            STATUS="failure"
          fi
          curl -X POST "${{ secrets.DISCORD_WEBHOOK }}" \
            -H 'Content-Type: application/json' \
            -d "{\"content\": \"Deploy $STATUS for commit ${{ github.sha }}\"}"

Lessons from Production

FTP is slow — The parallel matrix strategy cuts total deploy time from 12 minutes (sequential) to about 4 minutes.
Line endings matter — Config files must be Unix LF. A stray \r in FTP path configs corrupts remote paths silently.
Exclude patterns are critical — Never deploy .env files or data/ directories. One bad deploy can wipe your production database.

The pipeline has handled 200+ deploys to DailyWatch and the other sites without a manual intervention. It's not fancy, but it's reliable.

This article is part of the Building DailyWatch series.

CI/CD Pipeline for a Multi-Site Video Platform

ahmet gedik — Wed, 29 Apr 2026 09:00:01 +0000

The TrendVidStream codebase is shared across 4 LiteSpeed servers. Each server has its own .env with region assignments, API keys, and site-specific config. A git push to main should safely update all 4 without human intervention.

GitHub Actions handles this with a matrix deploy strategy.

Repository Layout Relevant to CI

v2/
├── app/
├── public/
├── templates/
├── cron/
├── site_envs/
│   ├── dwv.env
│   ├── tvh.env
│   ├── tvs.env        ← TrendVidStream production config
│   └── vvv.env
├── deploy_hosts.conf  ← FTP credentials (NEVER committed)
├── api_keys.conf      ← YouTube keys (committed, excluded from .gitignore)
├── version.txt        ← Git SHA, bumped on each deploy
└── .github/
    └── workflows/
        └── deploy.yml

Full Workflow

# .github/workflows/deploy.yml
name: CI/CD

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  PHP_VERSION: '8.3'

jobs:
  # ─── Phase 1: Static analysis + tests ───────────────────────
  quality:
    name: Quality Gates
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup PHP ${{ env.PHP_VERSION }}
        uses: shivammathur/setup-php@v2
        with:
          php-version: ${{ env.PHP_VERSION }}
          extensions: pdo_sqlite, gd, curl, intl
          tools: phpstan, cs2pr
          coverage: none

      - name: Cache Composer
        uses: actions/cache@v4
        with:
          path: vendor
          key: composer-${{ hashFiles('composer.lock') }}

      - name: Install dependencies
        run: composer install --no-dev --optimize-autoloader

      - name: PHP syntax check
        run: >
          find app/ public/ cron/ -name '*.php'
          -exec php -l {} \; 2>&1 | tee /tmp/syntax.log;
          grep -v 'No syntax errors' /tmp/syntax.log | grep -v '^$' || true

      - name: PHPStan analysis
        run: vendor/bin/phpstan analyse app/ --level=6 --no-progress

      - name: Unit tests
        run: vendor/bin/phpunit tests/ --testdox --colors=always

      - name: Validate deploy_hosts.conf line endings
        run: |
          if file deploy_hosts.conf | grep -q CRLF; then
            echo "::error::deploy_hosts.conf has Windows line endings"
            exit 1
          fi

      - name: Validate .env files exist for all sites
        run: |
          for site in dwv tvh tvs vvv; do
            if [ ! -f "site_envs/${site}.env" ]; then
              echo "::error::Missing site_envs/${site}.env"
              exit 1
            fi
          done
          echo "All .env files present"

      - name: Check no secrets in tracked files
        run: |
          # Ensure .env files are not tracked
          if git ls-files | grep -q '\.env$'; then
            echo "::error::.env file is tracked by git"
            exit 1
          fi

  # ─── Phase 2: Write version.txt ──────────────────────────────
  version:
    name: Tag Version
    needs: quality
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Write version file
        run: |
          echo "${{ github.sha }}" > version.txt
          echo "$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> version.txt

      - name: Upload version artifact
        uses: actions/upload-artifact@v4
        with:
          name: version
          path: version.txt

  # ─── Phase 3: Parallel deploy to all 4 hosts ─────────────────
  deploy:
    name: Deploy — ${{ matrix.site }}
    needs: version
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        include:
          - site: tvs
            site_env: tvs.env
          - site: dwv
            site_env: dwv.env
          - site: tvh
            site_env: tvh.env
          - site: vvv
            site_env: vvv.env

    steps:
      - uses: actions/checkout@v4

      - name: Download version artifact
        uses: actions/download-artifact@v4
        with:
          name: version

      - name: Install lftp
        run: sudo apt-get install -y lftp

      - name: Deploy to ${{ matrix.site }}
        env:
          FTP_HOST: ${{ secrets[format('FTP_{0}_HOST', matrix.site)] }}
          FTP_USER: ${{ secrets[format('FTP_{0}_USER', matrix.site)] }}
          FTP_PASS: ${{ secrets[format('FTP_{0}_PASS', matrix.site)] }}
          FTP_PATH: ${{ secrets[format('FTP_{0}_PATH', matrix.site)] }}
        run: |
          lftp -u "$FTP_USER","$FTP_PASS" "$FTP_HOST" << 'LFTP'
            set ssl:verify-certificate no
            set net:timeout 60
            set net:max-retries 5
            set net:reconnect-interval-base 5

            # Mirror app files (excludes data, logs, .env)
            mirror -R --ignore-time --verbose \
              --exclude .git/ \
              --exclude .github/ \
              --exclude backlink/ \
              --exclude 'site_envs/' \
              --exclude 'data/' \
              --exclude '*.log' \
              --exclude '*.db' \
              ./ $FTP_PATH

            # Deploy this site's env file
            put site_envs/${{ matrix.site_env }} -o $FTP_PATH/.env

            # Deploy shared API keys
            put api_keys.conf -o $FTP_PATH/api_keys.conf

            # Deploy version marker
            put version.txt -o $FTP_PATH/version.txt

            # Clear LiteSpeed page cache
            rm -rf $FTP_PATH/lscache

            quit
          LFTP

      - name: Verify deployment
        env:
          FTP_HOST: ${{ secrets[format('FTP_{0}_HOST', matrix.site)] }}
          FTP_USER: ${{ secrets[format('FTP_{0}_USER', matrix.site)] }}
          FTP_PASS: ${{ secrets[format('FTP_{0}_PASS', matrix.site)] }}
          FTP_PATH: ${{ secrets[format('FTP_{0}_PATH', matrix.site)] }}
        run: |
          REMOTE=$(lftp -u "$FTP_USER","$FTP_PASS" "$FTP_HOST" -e \
            "set ssl:verify-certificate no; get $FTP_PATH/version.txt -o /tmp/remote_ver.txt; quit")
          LOCAL=$(head -1 version.txt)
          REMOTE_SHA=$(head -1 /tmp/remote_ver.txt)
          if [ "$LOCAL" = "$REMOTE_SHA" ]; then
            echo "[${{ matrix.site }}] OK: $REMOTE_SHA"
          else
            echo "[${{ matrix.site }}] MISMATCH: local=$LOCAL remote=$REMOTE_SHA"
            exit 1
          fi

  # ─── Phase 4: Summary ─────────────────────────────────────────
  summary:
    name: Deploy Summary
    needs: deploy
    if: always()
    runs-on: ubuntu-latest
    steps:
      - name: Output result
        run: |
          echo "## Deploy Result" >> $GITHUB_STEP_SUMMARY
          echo "| Site | Status |" >> $GITHUB_STEP_SUMMARY
          echo "|---|---|" >> $GITHUB_STEP_SUMMARY
          echo "| tvs | ${{ needs.deploy.result }} |" >> $GITHUB_STEP_SUMMARY
          echo "| dwv | ${{ needs.deploy.result }} |" >> $GITHUB_STEP_SUMMARY
          echo "| tvh | ${{ needs.deploy.result }} |" >> $GITHUB_STEP_SUMMARY
          echo "| vvv | ${{ needs.deploy.result }} |" >> $GITHUB_STEP_SUMMARY

Pull Request Workflow (Test Only)

On pull requests, only the quality gates run — no deployment:

# Reuses the same quality job, skips version + deploy
# because of the `if: github.ref == 'refs/heads/main'` guards

Cron Monitoring Job

# .github/workflows/monitor.yml
name: Monitor Cron Health

on:
  schedule:
    - cron: '0 6 * * *'  # Daily at 06:00 UTC

jobs:
  ping:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        url:
          - https://trendvidstream.com/health
          - https://topvideohub.com/health
          - https://dailywatch.video/health
    steps:
      - name: Health check ${{ matrix.url }}
        run: |
          STATUS=$(curl -s -o /dev/null -w '%{http_code}' \
            --max-time 10 '${{ matrix.url }}')
          if [ "$STATUS" != "200" ]; then
            echo "::error::${{ matrix.url }} returned HTTP $STATUS"
            exit 1
          fi
          echo "${{ matrix.url }} OK ($STATUS)"

Total Pipeline Time

Phase	Time
Quality gates	~50s
Version tag	~5s
Parallel FTP × 4	~3.5min
Verification × 4	~20s
Total	~5min

Every commit to TrendVidStream now deploys to all 4 global servers in under 5 minutes with zero manual steps.

This is part of the "Building TrendVidStream" series, documenting the architecture behind a global video directory covering Nordic, Middle Eastern, and Central European regions.

CI/CD Pipeline for a Multi-Site Video Platform

ahmet gedik — Tue, 28 Apr 2026 23:00:01 +0000

ViralVidVault is part of a family of video platforms, each running on separate LiteSpeed shared hosting. Same codebase, different configurations. Deploying manually to multiple hosts was error-prone and tedious. Here's the GitHub Actions pipeline that replaced it.

Constraints

Shared hosting means no SSH, no Docker on the server, no git pull. The only deployment option is FTP. The pipeline needs to:

Run PHP linting and tests
Deploy to multiple hosts in parallel
Exclude sensitive files (.env, data/)
Clear LiteSpeed cache after deploy
Verify each deployment succeeded

The Workflow

# .github/workflows/deploy.yml
name: Test and Deploy

on:
  push:
    branches: [main]
  workflow_dispatch:
    inputs:
      site:
        description: 'Deploy specific site (or all)'
        required: false
        default: 'all'
        type: choice
        options: [all, dwv, tvh, vvv, tvs]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: shivammathur/setup-php@v2
        with:
          php-version: '8.3'
          extensions: sqlite3, pdo_sqlite, curl
      - name: Lint PHP files
        run: |
          EXIT_CODE=0
          while IFS= read -r file; do
            php -l "$file" > /dev/null 2>&1 || EXIT_CODE=1
          done < <(find app/ public/ cron/ -name '*.php')
          exit $EXIT_CODE
      - name: Run test suite
        run: php tests/run.php
        env:
          APP_ENV: testing

  deploy:
    needs: test
    runs-on: ubuntu-latest
    strategy:
      matrix:
        site: [dwv, tvh, vvv, tvs]
      fail-fast: false
    steps:
      - uses: actions/checkout@v4

      - name: Deploy via FTP
        uses: SamKirkland/FTP-Deploy-Action@v4.3.5
        with:
          server: ${{ secrets[format('{0}_FTP_HOST', matrix.site)] }}
          username: ${{ secrets[format('{0}_FTP_USER', matrix.site)] }}
          password: ${{ secrets[format('{0}_FTP_PASS', matrix.site)] }}
          server-dir: /public_html/
          exclude: |
            **/.env
            **/data/**
            **/*.log
            **/.git/**
            **/tests/**
            **/backlink/**

      - name: Clear LiteSpeed cache
        run: |
          curl -sf "${{ secrets[format('{0}_URL', matrix.site)] }}/task/clear-cache?key=${{ secrets.TASK_KEY }}" \
            --max-time 10 || echo "Cache clear timed out (non-fatal)"

The workflow_dispatch input lets you deploy a single site manually from the GitHub UI. Useful when you need to hotfix one site without touching the others.

Matrix Strategy Deep Dive

The matrix keyword creates four parallel jobs, one per site. Each job has its own runner and its own set of secrets:

    strategy:
      matrix:
        site: [dwv, tvh, vvv, tvs]
      fail-fast: false

fail-fast: false is critical. Without it, if the tvh deploy fails (say, FTP timeout), GitHub cancels the vvv, dwv, and tvs jobs too. That turns one problem into four.

Secrets Organization

Twelve repository secrets, three per site:

DWV_FTP_HOST=ftp.dailywatch.video
DWV_FTP_USER=deploy@dailywatch.video
DWV_FTP_PASS=<password>

VVV_FTP_HOST=ftp.viralvidvault.com
VVV_FTP_USER=deploy@viralvidvault.com
VVV_FTP_PASS=<password>

# ... same pattern for TVH, TVS

The format() function dynamically resolves the right secret:

server: ${{ secrets[format('{0}_FTP_HOST', matrix.site)] }}
# When matrix.site = 'vvv', resolves to secrets.VVV_FTP_HOST

Post-Deploy Verification

Deploy doesn't mean working. Add a verification step:

      - name: Verify deployment
        run: |
          URL="${{ secrets[format('{0}_URL', matrix.site)] }}"
          echo "Checking $URL..."

          # Basic availability check
          HTTP_CODE=$(curl -s -o /dev/null -w '%{http_code}' "$URL" --max-time 15)
          if [ "$HTTP_CODE" != "200" ]; then
            echo "::error::${{ matrix.site }} returned HTTP $HTTP_CODE"
            exit 1
          fi

          # Version check via health endpoint
          DEPLOYED=$(curl -sf "$URL/health" --max-time 10 | python3 -c "import sys,json; print(json.load(sys.stdin).get('commit','unknown'))")
          echo "${{ matrix.site }}: HTTP $HTTP_CODE, version $DEPLOYED"

This catches silent failures — FTP uploads that seem to succeed but actually fail due to permissions, disk space, or path misconfigurations.

Notifications on Failure

A deploy that fails silently is worse than a deploy that fails loudly:

  notify:
    needs: deploy
    if: failure()
    runs-on: ubuntu-latest
    steps:
      - name: Alert on failure
        run: |
          curl -X POST "${{ secrets.DISCORD_WEBHOOK }}" \
            -H 'Content-Type: application/json' \
            -d '{"content": "Deploy FAILED for ${{ github.sha }}. Check: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"}'

Production Results

Since switching to this pipeline for ViralVidVault and the other sites:

Deploy time went from 15 minutes (manual, sequential) to 4 minutes (automated, parallel)
Zero missed deployments (no more "forgot to deploy to tvs")
Three bad deploys caught by the verification step before any user noticed

The FTP constraint makes this less elegant than a container-based deploy, but the reliability improvement is the same. Automate the boring parts, verify the important parts.

This article is part of the Building ViralVidVault series.

CI/CD Pipeline for a Multi-Site Video Platform

ahmet gedik — Tue, 28 Apr 2026 13:13:14 +0000

Deploying TopVideoHub means pushing to 4 LiteSpeed servers that each cover different Asia-Pacific regions. Manual FTP to 4 hosts is error-prone. GitHub Actions turned that into a one-push workflow.

Here is the full pipeline.

Constraints: Shared Hosting Reality

Shared LiteSpeed hosting does not give you SSH access, Docker runners on the server, or webhook endpoints. The deployment method is FTP via lftp. The CI pipeline must:

Run tests and lint PHP without a server
Deploy via FTP in parallel to all 4 hosts
Clear the LiteSpeed page cache after deploy
Send a notification when all hosts are live

Repository Secrets

In GitHub Settings → Secrets and variables → Actions, create one secret per host:

FTP_DWV_HOST=ftp.dailywatch.video
FTP_DWV_USER=deploy_user
FTP_DWV_PASS=super_secret
FTP_DWV_PATH=/htdocs

FTP_TVH_HOST=ftp.topvideohub.com
FTP_TVH_USER=deploy_user
FTP_TVH_PASS=super_secret
FTP_TVH_PATH=/htdocs

# ... repeat for tvs and vvv

Main Workflow

# .github/workflows/deploy.yml
name: Test and Deploy

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  # ─── Phase 1: Test ───────────────────────────────────────────
  test:
    name: PHP Tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup PHP
        uses: shivammathur/setup-php@v2
        with:
          php-version: '8.3'
          extensions: pdo_sqlite, gd, curl
          coverage: none

      - name: Validate composer.json
        run: composer validate --strict

      - name: Install dependencies
        run: composer install --no-dev --optimize-autoloader

      - name: PHP Lint (syntax check)
        run: |
          find app/ public/ cron/ templates/ \
            -name '*.php' \
            -exec php -l {} \; | grep -v 'No syntax errors'

      - name: Run unit tests
        run: vendor/bin/phpunit tests/ --testdox

      - name: Validate .htaccess syntax
        run: |
          if grep -P '\t' public/.htaccess; then
            echo "ERROR: .htaccess contains tab characters"
            exit 1
          fi
          echo ".htaccess OK"

      - name: Check deploy_hosts.conf line endings
        run: |
          if grep -Pc '\r' deploy_hosts.conf > /dev/null 2>&1; then
            echo "ERROR: deploy_hosts.conf has Windows line endings!"
            exit 1
          fi
          echo "deploy_hosts.conf OK"

  # ─── Phase 2: Deploy (only on push to main) ───────────────────
  deploy:
    name: Deploy — ${{ matrix.site }}
    needs: test
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false          # Don't cancel other deploys if one fails
      matrix:
        site: [dwv, tvh, tvs, vvv]

    steps:
      - uses: actions/checkout@v4

      - name: Install lftp
        run: sudo apt-get install -y lftp

      - name: Deploy via FTP
        env:
          FTP_HOST: ${{ secrets[format('FTP_{0}_HOST', matrix.site)] }}
          FTP_USER: ${{ secrets[format('FTP_{0}_USER', matrix.site)] }}
          FTP_PASS: ${{ secrets[format('FTP_{0}_PASS', matrix.site)] }}
          FTP_PATH: ${{ secrets[format('FTP_{0}_PATH', matrix.site)] }}
        run: |
          lftp -u "$FTP_USER","$FTP_PASS" "$FTP_HOST" << 'LFTP'
            set ssl:verify-certificate no
            set net:timeout 60
            set net:max-retries 3
            mirror -R --verbose --ignore-time \
              --exclude .git/ \
              --exclude .github/ \
              --exclude .env \
              --exclude 'data/' \
              --exclude '*.log' \
              --exclude '*.db' \
              --exclude 'backlink/' \
              ./ $FTP_PATH
            put api_keys.conf -o $FTP_PATH/api_keys.conf
            rm -rf $FTP_PATH/lscache
            quit
          LFTP

      - name: Verify deploy
        env:
          FTP_HOST: ${{ secrets[format('FTP_{0}_HOST', matrix.site)] }}
          FTP_USER: ${{ secrets[format('FTP_{0}_USER', matrix.site)] }}
          FTP_PASS: ${{ secrets[format('FTP_{0}_PASS', matrix.site)] }}
          FTP_PATH: ${{ secrets[format('FTP_{0}_PATH', matrix.site)] }}
        run: |
          REMOTE_VER=$(lftp -u "$FTP_USER","$FTP_PASS" "$FTP_HOST" -e \
            "set ssl:verify-certificate no; cat $FTP_PATH/version.txt; quit")
          LOCAL_VER=$(cat version.txt)
          if [ "$REMOTE_VER" = "$LOCAL_VER" ]; then
            echo "[${{ matrix.site }}] Verify PASSED: $REMOTE_VER"
          else
            echo "[${{ matrix.site }}] Verify FAILED: remote=$REMOTE_VER local=$LOCAL_VER"
            exit 1
          fi

  # ─── Phase 3: Post-deploy notification ───────────────────────
  notify:
    name: Notify
    needs: deploy
    if: always()
    runs-on: ubuntu-latest
    steps:
      - name: Post status to Slack
        uses: slackapi/slack-github-action@v1.26.0
        with:
          payload: |
            {
              "text": "TopVideoHub deploy ${{ needs.deploy.result == 'success' && '✓ succeeded' || '✗ failed' }} on `${{ github.ref_name }}` by ${{ github.actor }}"
            }
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}

PHP Unit Test Example

<?php
// tests/QuotaManagerTest.php

use PHPUnit\Framework\TestCase;

class QuotaManagerTest extends TestCase
{
    private PDO $db;

    protected function setUp(): void
    {
        // In-memory SQLite for tests — no file I/O
        $this->db = new PDO('sqlite::memory:');
        $this->db->exec(file_get_contents(__DIR__ . '/../schema.sql'));
    }

    public function testQuotaDecrement(): void
    {
        $mgr = new QuotaManager($this->db);
        $mgr->setQuota('key1', 10000);

        $mgr->consume('key1', 100);
        $this->assertSame(9900, $mgr->remaining('key1'));
    }

    public function testKeyRotation(): void
    {
        $mgr = new QuotaManager($this->db);
        $mgr->setQuota('key1', 0);   // Exhausted
        $mgr->setQuota('key2', 10000);

        $activeKey = $mgr->getActiveKey();
        $this->assertSame('key2', $activeKey);
    }

    public function testRegionFetchOrder(): void
    {
        $regions = ['US', 'GB', 'JP', 'KR', 'TW', 'SG', 'VN', 'TH', 'HK'];
        $this->assertCount(9, $regions);
        $this->assertContains('JP', $regions);
        $this->assertContains('KR', $regions);
    }
}

Caching the Composer Install

- name: Cache Composer packages
  uses: actions/cache@v4
  with:
    path: vendor
    key: composer-${{ hashFiles('composer.lock') }}
    restore-keys: composer-

- name: Install dependencies
  run: composer install --no-dev --optimize-autoloader

Parallel Deploy Performance

The matrix strategy runs all 4 deployments simultaneously. Wall-clock time for a full deploy to all 4 LiteSpeed hosts:

Step	Time
PHP tests + lint	~45s
FTP mirror (parallel × 4)	~3min
Verify + notify	~15s
Total	~4min

Without parallelism, sequential FTP to 4 hosts would take ~12 minutes.

Branch Protection

# .github/branch_protection.yml (via GitHub API or UI)
branch: main
required_status_checks:
  strict: true
  contexts:
    - "PHP Tests"
require_pull_request_reviews:
  required_approving_review_count: 1
restrict_pushes: true

Every commit to TopVideoHub now goes through this pipeline. The matrix deploy means all 4 servers are updated in under 4 minutes from a git push.

This is part of the "Building TopVideoHub" series, documenting the architecture behind a video discovery platform covering 9 Asia-Pacific regions.

Redis Caching Strategies for Video Content Platforms

ahmet gedik — Mon, 20 Apr 2026 05:00:02 +0000

TrendVidStream serves trending video content from 8 regions: US, GB, CH, DK, AE, BE, CZ, FI. The Nordic and Central European regions trend quietly, while UAE can spike hard when regional events drive traffic. A flat TTL for all regions wastes quota on quiet regions and serves stale data to spiking ones.

Redis lets us handle that heterogeneity cleanly.

Current Stack vs Redis

The platform currently uses a 3-tier PHP file cache. Redis fits in as the middle tier:

Request
  → Redis (sub-millisecond, in-memory)
  → PHP file cache (milliseconds, disk)
  → SQLite (single-digit ms, WAL mode)

For a multi-worker PHP-FPM setup, Redis solves the shared-state problem: file-based caches are per-server. Redis is shared across all workers automatically.

Trending Lists: Redis Lists

<?php

class TrendingCache
{
    private Redis $r;

    // Region-specific TTLs — Nordic/Central EU trends slowly,
    // UAE and GB can spike fast
    private const REGION_TTL = [
        'AE' => 5400,   // 1.5h — UAE news/events spike
        'GB' => 7200,   // 2h
        'US' => 7200,   // 2h
        'CH' => 10800,  // 3h — stable Swiss market
        'DK' => 10800,  // 3h — stable Nordic
        'BE' => 10800,  // 3h
        'CZ' => 14400,  // 4h — Central EU slower cycle
        'FI' => 14400,  // 4h — Finland quietest
    ];

    public function set(string $region, array $videos): void
    {
        $key = "tvs:trending:{$region}";
        $ttl = self::REGION_TTL[$region] ?? 10800;

        $pipe = $this->r->pipeline();
        $pipe->del($key);

        foreach ($videos as $video) {
            $pipe->rPush($key, json_encode([
                'id'        => $video['video_id'],
                'title'     => $video['title'],
                'thumb'     => $video['thumbnail'],
                'channel'   => $video['channel_title'],
                'views'     => $video['view_count'],
                'category'  => $video['category_id'],
            ]));
        }

        $pipe->expire($key, $ttl);
        $pipe->execute();
    }

    public function get(string $region, int $page = 1, int $perPage = 20): ?array
    {
        $key   = "tvs:trending:{$region}";
        $start = ($page - 1) * $perPage;
        $end   = $start + $perPage - 1;

        $items = $this->r->lRange($key, $start, $end);
        if (empty($items)) {
            return null;
        }

        return array_map(
            fn($v) => json_decode($v, true),
            $items
        );
    }

    public function count(string $region): int
    {
        return (int)$this->r->lLen("tvs:trending:{$region}");
    }
}

Sorted Sets for Cross-Region Trending

TrendVidStream's home page surfaces videos trending in the most regions simultaneously — a pan-European trending feed:

<?php

class CrossRegionCache
{
    /**
     * Score = number of regions a video is trending in.
     * ZUNIONSTORES the region lists into a single cross-region feed.
     */
    public function rebuild(array $regions): void
    {
        $tempKeys = [];

        foreach ($regions as $region) {
            $srcKey  = "tvs:trending:{$region}";
            $tempKey = "tvs:temp:scores:{$region}";

            // Fetch all video IDs from this region's list
            $videos = $this->r->lRange($srcKey, 0, -1);
            if (empty($videos)) {
                continue;
            }

            $pipe = $this->r->pipeline();
            $pipe->del($tempKey);
            foreach ($videos as $raw) {
                $v = json_decode($raw, true);
                $pipe->zAdd($tempKey, 1, $v['id']);
            }
            $pipe->execute();

            $tempKeys[] = $tempKey;
        }

        if (empty($tempKeys)) {
            return;
        }

        // Union all region scores — video trending in 4 regions scores 4
        $this->r->zUnionStore(
            'tvs:global:trending',
            $tempKeys,
            array_fill(0, count($tempKeys), 1),
            'SUM'
        );
        $this->r->expire('tvs:global:trending', 10800);

        // Cleanup temp keys
        $this->r->del($tempKeys);
    }

    public function getTopGlobal(int $n = 20): array
    {
        // Highest score = most regions trending
        return $this->r->zRevRange('tvs:global:trending', 0, $n - 1, true);
    }
}

Hash Sets for Video Metadata

<?php

class VideoMetaCache
{
    private const TTL = 21600; // 6h — matches watch page HTTP cache

    public function set(string $videoId, array $meta): void
    {
        $key = "tvs:video:{$videoId}";
        $this->r->hMSet($key, [
            'title'       => $meta['title'],
            'description' => mb_substr($meta['description'], 0, 500),
            'channel'     => $meta['channel_title'],
            'thumb_high'  => $meta['thumbnail_high'],
            'view_count'  => $meta['view_count'],
            'like_count'  => $meta['like_count'],
            'category_id' => $meta['category_id'],
            'region'      => $meta['region'],
            'cached_at'   => time(),
        ]);
        $this->r->expire($key, self::TTL);
    }

    public function get(string $videoId): ?array
    {
        $data = $this->r->hGetAll("tvs:video:{$videoId}");
        return empty($data) ? null : $data;
    }

    public function mGet(array $videoIds): array
    {
        $pipe = $this->r->pipeline();
        foreach ($videoIds as $id) {
            $pipe->hGetAll("tvs:video:{$id}");
        }
        $results = $pipe->execute();

        return array_combine(
            $videoIds,
            array_map(fn($r) => empty($r) ? null : $r, $results)
        );
    }
}

Cache Invalidation Patterns

Write-Through on Cron Fetch

<?php

class CronFetcher
{
    public function fetchRegion(string $region): void
    {
        $videos = $this->youtube->fetchTrending($region);

        // 1. Persist to SQLite
        $this->db->upsertVideos($videos, $region);

        // 2. Write-through: update Redis immediately
        $this->trendingCache->set($region, $videos);

        // 3. Invalidate search results for this region
        $this->invalidateSearchCache($region);

        // 4. Invalidate PHP page cache for affected pages
        $this->clearPageCache($region);
    }

    private function invalidateSearchCache(string $region): void
    {
        // Use SCAN — never KEYS in production
        $cursor = null;
        do {
            [$cursor, $keys] = $this->r->scan(
                $cursor,
                ['match' => "tvs:search:{$region}:*", 'count' => 200]
            );
            if ($keys) {
                $this->r->del($keys);
            }
        } while ($cursor !== 0);
    }
}

Stale-While-Revalidate in PHP

<?php

function getTrendingWithSWR(string $region, TrendingCache $cache, Database $db): array
{
    $data = $cache->get($region);

    if ($data !== null) {
        $ttl = $cache->getTTL($region);
        // If less than 20% of TTL remains, trigger async refresh
        if ($ttl < ($cache::REGION_TTL[$region] ?? 10800) * 0.2) {
            // Schedule background refresh without blocking the request
            if (function_exists('fastcgi_finish_request')) {
                register_shutdown_function(function() use ($region, $db, $cache) {
                    fastcgi_finish_request();
                    $fresh = $db->getTrending($region, 50);
                    $cache->set($region, $fresh);
                });
            }
        }
        return $data;
    }

    // Cache miss — fetch from SQLite
    $videos = $db->getTrending($region, 50);
    $cache->set($region, $videos);
    return array_slice($videos, 0, 20);
}

Memory Budget

For 8 regions at TrendVidStream:

Key type	Count	Avg size	Total
Trending lists	8 × 50 items	~600B/item	~240KB
Video metadata hashes	~400 videos	~900B	~360KB
Search results	~300 cached	~2KB	~600KB
Global trending sorted set	1	~15KB	~15KB
Total			~1.2MB

A 32MB Redis instance comfortably serves TrendVidStream with room for 25× growth.

This is part of the "Building TrendVidStream" series, documenting the architecture behind a global video directory covering Nordic, Middle Eastern, and Central European regions.