Rate Limiting and API Key Management for Video Data APIs

The YouTube Data API v3 gives you 10,000 quota units per day per key. That sounds generous until you realize a single search request costs 100 units and a video details call costs 1 unit per video. Do the math for a platform that fetches trending content from 8 regions every 2 hours, and you hit the wall fast. Here's how DailyWatch handles it.

Understanding YouTube API Quotas

Not all API calls are equal. The quota cost varies by endpoint:

Endpoint	Cost per call
search.list	100 units
videos.list	1 unit
channels.list	1 unit
playlistItems.list	1 unit

A single search.list call returning 50 results costs the same 100 units as one returning 5. Always request maxResults=50 to maximize value per call.

Multi-Key Rotation

One key gives 10,000 units. Three keys give 30,000. The rotation logic is straightforward:

class ApiKeyManager {
    private array $keys;
    private PDO $db;

    public function __construct(PDO $db) {
        $this->db = $db;
        $this->keys = $this->loadKeys();
    }

    private function loadKeys(): array {
        $stmt = $this->db->query(
            "SELECT api_key, daily_usage, last_reset 
             FROM api_keys 
             WHERE active = 1 
             ORDER BY daily_usage ASC"
        );
        return $stmt->fetchAll(PDO::FETCH_ASSOC);
    }

    public function getNextKey(): ?string {
        foreach ($this->keys as $key) {
            // Reset counter if new day (Pacific Time, matching Google's reset)
            if ($this->isNewDay($key['last_reset'])) {
                $this->resetUsage($key['api_key']);
                return $key['api_key'];
            }
            // Skip exhausted keys
            if ($key['daily_usage'] < 9500) { // 500 unit buffer
                return $key['api_key'];
            }
        }
        return null; // All keys exhausted
    }

    public function recordUsage(string $apiKey, int $units): void {
        $stmt = $this->db->prepare(
            "UPDATE api_keys 
             SET daily_usage = daily_usage + ?, last_used = datetime('now') 
             WHERE api_key = ?"
        );
        $stmt->execute([$units, $apiKey]);
    }
}

Keys are sorted by usage ascending, so the least-used key always gets picked first. The 500-unit buffer prevents accidentally exceeding the quota on the last call of the day.

Rate Limiting the Fetch Process

Even with multiple keys, you don't want to fire 50 API calls in a tight loop. Google enforces per-second rate limits too:

class RateLimiter {
    private float $lastCallTime = 0;
    private float $minInterval;

    public function __construct(float $callsPerSecond = 5.0) {
        $this->minInterval = 1.0 / $callsPerSecond;
    }

    public function wait(): void {
        $elapsed = microtime(true) - $this->lastCallTime;
        if ($elapsed < $this->minInterval) {
            usleep((int)(($this->minInterval - $elapsed) * 1_000_000));
        }
        $this->lastCallTime = microtime(true);
    }
}

// Usage in fetch loop
$limiter = new RateLimiter(3.0); // 3 calls per second
$keyManager = new ApiKeyManager($db);

foreach ($regions as $region) {
    $limiter->wait();
    $apiKey = $keyManager->getNextKey();
    if (!$apiKey) {
        log_message('All API keys exhausted, stopping fetch');
        break;
    }
    $results = fetchTrending($apiKey, $region);
    $keyManager->recordUsage($apiKey, 100); // search cost
}

Quota-Efficient Fetching

The biggest optimization is reducing search calls. Instead of searching per category per region, batch smartly:

// BAD: 8 regions x 15 categories = 120 search calls = 12,000 units
foreach ($regions as $region) {
    foreach ($categories as $category) {
        searchVideos($region, $category); // 100 units each
    }
}

// GOOD: Use chart=mostPopular (1 unit) + videoCategoryId filter
foreach ($regions as $region) {
    // 1 unit instead of 100!
    $popular = getPopularVideos($apiKey, $region, 50);
    foreach ($categories as $category) {
        $catVideos = getPopularVideos($apiKey, $region, 50, $category);
    }
}

The videos.list endpoint with chart=mostPopular costs 1 unit versus 100 for search.list. For 8 regions with 15 categories, that's 128 units instead of 12,000.

Handling 403 Quota Exceeded

When a key hits its limit, Google returns HTTP 403 with a specific error reason. Handle it gracefully:

function makeApiCall(ApiKeyManager $km, string $url): ?array {
    $maxRetries = count($km->getKeys());
    for ($i = 0; $i < $maxRetries; $i++) {
        $key = $km->getNextKey();
        if (!$key) return null;

        $response = httpGet($url . '&key=' . $key);

        if ($response['status'] === 200) {
            return json_decode($response['body'], true);
        }

        if ($response['status'] === 403) {
            $error = json_decode($response['body'], true);
            $reason = $error['error']['errors'][0]['reason'] ?? '';

            if ($reason === 'quotaExceeded' || $reason === 'dailyLimitExceeded') {
                $km->markExhausted($key);
                continue; // Try next key
            }
        }
    }
    return null;
}

The loop automatically rotates to the next available key on quota errors. If all keys are exhausted, it returns null and the caller decides whether to retry later.

Monitoring and Alerts

Track daily usage in the database and alert when approaching limits:

function checkQuotaHealth(PDO $db): array {
    $stmt = $db->query(
        "SELECT api_key, daily_usage,
                ROUND(daily_usage * 100.0 / 10000, 1) AS usage_pct
         FROM api_keys WHERE active = 1"
    );
    $keys = $stmt->fetchAll(PDO::FETCH_ASSOC);
    $totalUsed = array_sum(array_column($keys, 'daily_usage'));
    $totalAvailable = count($keys) * 10000;

    return [
        'keys_active'    => count($keys),
        'total_used'     => $totalUsed,
        'total_available' => $totalAvailable,
        'usage_percent'  => round($totalUsed / $totalAvailable * 100, 1),
        'per_key'        => $keys,
    ];
}

On DailyWatch, this data feeds into a simple admin dashboard. When combined usage crosses 80%, the system automatically reduces fetch frequency for lower-priority regions.

The key insight: treat API quotas as a finite resource to be budgeted, not a limit to be worked around. Design your fetch strategy around the quota, not the other way around.

---

This article is part of the Building DailyWatch series.