DEV Community

ahmet gedik
ahmet gedik

Posted on

Building a Multi-Region Cron System for Video Freshness

#php #cron #devops #webdev

Introduction

Keeping video content fresh across multiple regions requires a well-designed cron system. At ViralVidVault, a single cron command handles fetching trending videos from 7 European regions, refreshing stale content, and cleaning up old records. Here's how it all fits together.

The Cron Architecture

One script, multiple steps, executed sequentially:

<?php
// cron/fetch_videos.php

declare(strict_types=1);

require_once __DIR__ . '/../vendor/autoload.php';

$startTime = microtime(true);
$db = new Database(__DIR__ . '/../data/videos.db');
$youtube = new YouTubeClient($_ENV['YOUTUBE_API_KEY']);
$logger = new CronLogger(__DIR__ . '/../data/cron.log');

$regions = explode(',', $_ENV['FETCH_REGIONS'] ?? 'US,GB,PL,NL,SE,NO,AT');

try {
    // STEP 1: Fetch globally popular videos
    $logger->info('Step 1: Fetching popular videos...');
    $popular = $youtube->getPopular(maxResults: 50);
    $db->upsertVideos($popular);
    $logger->info("Step 1 complete: " . count($popular) . " popular videos");

    // STEP 2: Sync categories
    $logger->info('Step 2: Syncing categories...');
    $categories = $youtube->getCategories('US');
    $db->syncCategories($categories);
    $logger->info("Step 2 complete: " . count($categories) . " categories");

    // STEP 3: Regional trending loop
    $logger->info('Step 3: Fetching regional trending...');
    $totalRegional = 0;
    foreach ($regions as $region) {
        $region = trim($region);
        $trending = $youtube->getTrending($region, maxResults: 25);
        $db->upsertVideos($trending, region: $region);
        $totalRegional += count($trending);
        $logger->info("  [{$region}] " . count($trending) . " videos");
        usleep(300000); // 300ms between regions
    }
    $logger->info("Step 3 complete: {$totalRegional} regional videos");

    // STEP 4: Refresh stale content
    $logger->info('Step 4: Checking stale content...');
    $checker = new FreshnessChecker($db->getPdo(), $youtube);
    $freshResult = $checker->checkBatch(batchSize: 50);
    $logger->info("Step 4 complete: {$freshResult->staleCount} stale, {$freshResult->freshCount} fresh");

    // STEP 5: Cleanup old content
    $logger->info('Step 5: Cleaning up...');
    $cleaner = new ContentCleaner($db->getPdo());
    $cleanReport = $cleaner->run();
    $logger->info("Step 5 complete: {$cleanReport}");

    // STEP 6: Rebuild search index
    $logger->info('Step 6: Rebuilding search index...');
    $db->rebuildSearchIndex();
    $logger->info('Step 6 complete');

} catch (\Throwable $e) {
    $logger->error("Cron failed: {$e->getMessage()}");
    $logger->error($e->getTraceAsString());
    exit(1);
}

$elapsed = round(microtime(true) - $startTime, 1);
$logger->info("Cron complete in {$elapsed}s");
Enter fullscreen mode Exit fullscreen mode

The Logger 

<?php

class CronLogger
{
    private string $logFile;

    public function __construct(string $logFile)
    {
        $this->logFile = $logFile;
    }

    public function info(string $message): void
    {
        $this->write('INFO', $message);
    }

    public function error(string $message): void
    {
        $this->write('ERROR', $message);
    }

    private function write(string $level, string $message): void
    {
        $timestamp = date('Y-m-d H:i:s');
        $line = "[{$timestamp}] [{$level}] {$message}" . PHP_EOL;
        file_put_contents($this->logFile, $line, FILE_APPEND | LOCK_EX);
        echo $line; // Also output to stdout for cron email
    }
}
Enter fullscreen mode Exit fullscreen mode

Crontab Configuration

Different sites can run at different intervals:

# ViralVidVault - every 7 hours
35 */7 * * * cd /home/user/viralvidvault && php cron/fetch_videos.php >> data/cron_output.log 2>&1
Enter fullscreen mode Exit fullscreen mode

Environment-Based Region Configuration

Each deployment defines its own regions via environment variables:

# .env for viralvidvault.com
FETCH_REGIONS=US,GB,PL,NL,SE,NO,AT
YOUTUBE_API_KEY=your_key_here
Enter fullscreen mode Exit fullscreen mode 
// Reading regions from environment
$regions = array_map('trim', explode(',', $_ENV['FETCH_REGIONS'] ?? 'US,GB'));
Enter fullscreen mode Exit fullscreen mode

Handling API Quota Across Regions

With 7 regions and various API calls per cron run, quota management matters:

<?php

class QuotaTracker
{
    private int $used = 0;
    private int $daily_limit;

    public function __construct(int $dailyLimit = 10000)
    {
        $this->daily_limit = $dailyLimit;
    }

    public function track(string $operation, int $cost): void
    {
        $this->used += $cost;
    }

    public function canAfford(int $cost): bool
    {
        return ($this->used + $cost) <= $this->daily_limit;
    }

    public function remaining(): int
    {
        return max(0, $this->daily_limit - $this->used);
    }
}

// Usage in cron
$quota = new QuotaTracker(dailyLimit: 10000);

foreach ($regions as $region) {
    if (!$quota->canAfford(2)) { // 1 for trending + 1 for stats
        $logger->info("Quota limit reached, skipping remaining regions");
        break;
    }

    $trending = $youtube->getTrending($region);
    $quota->track('trending', 1);

    $db->upsertVideos($trending, region: $region);
}

$logger->info("Quota used: {$quota->remaining()} remaining");
Enter fullscreen mode Exit fullscreen mode

Monitoring Cron Health

A simple health check endpoint:

// Route: /api/health
$lastCron = $db->query('SELECT MAX(fetched_at) as last FROM videos')->fetchColumn();
$hoursSince = (time() - strtotime($lastCron)) / 3600;

$healthy = $hoursSince < 24; // Alert if no fetch in 24h

header('Content-Type: application/json');
echo json_encode([
    'status' => $healthy ? 'ok' : 'stale',
    'last_fetch' => $lastCron,
    'hours_since' => round($hoursSince, 1),
    'video_count' => $db->query('SELECT COUNT(*) FROM videos WHERE is_active=1')->fetchColumn(),
]);
Enter fullscreen mode Exit fullscreen mode

Real-World Stats

On viralvidvault.com, a typical cron run:

[2026-03-02 14:35:01] [INFO] Step 1: Fetching popular videos...
[2026-03-02 14:35:03] [INFO] Step 1 complete: 50 popular videos
[2026-03-02 14:35:03] [INFO] Step 2: Syncing categories...
[2026-03-02 14:35:04] [INFO] Step 2 complete: 18 categories
[2026-03-02 14:35:04] [INFO] Step 3: Fetching regional trending...
[2026-03-02 14:35:05] [INFO]   [US] 25 videos
[2026-03-02 14:35:06] [INFO]   [GB] 25 videos
[2026-03-02 14:35:06] [INFO]   [PL] 25 videos
[2026-03-02 14:35:07] [INFO]   [NL] 25 videos
[2026-03-02 14:35:08] [INFO]   [SE] 25 videos
[2026-03-02 14:35:08] [INFO]   [NO] 25 videos
[2026-03-02 14:35:09] [INFO]   [AT] 25 videos
[2026-03-02 14:35:09] [INFO] Step 3 complete: 175 regional videos
[2026-03-02 14:35:09] [INFO] Step 4: Checking stale content...
[2026-03-02 14:35:11] [INFO] Step 4 complete: 3 stale, 47 fresh
[2026-03-02 14:35:11] [INFO] Step 5: Cleaning up...
[2026-03-02 14:35:12] [INFO] Step 5 complete: Cleanup: 22 removed
[2026-03-02 14:35:12] [INFO] Step 6: Rebuilding search index...
[2026-03-02 14:35:13] [INFO] Step 6 complete
[2026-03-02 14:35:13] [INFO] Cron complete in 12.1s
Enter fullscreen mode Exit fullscreen mode

12 seconds for a full 7-region fetch, stale check, cleanup, and indexing. SQLite and PHP handle it with ease.

Key Takeaways

  1. One cron script, sequential steps — keeps it simple and debuggable
  2. Use environment variables for region configuration
  3. Add rate limiting between region fetches
  4. Track API quota to avoid exceeding limits
  5. Log everything for post-mortem debugging
  6. Build a health check endpoint for monitoring

Part of the "Building ViralVidVault" series. Explore the results at ViralVidVault.

Top comments (0)