DEV Community

Cover image for How I Built a YouTube Trend Engine on Zerops
Sagar Maurya
Sagar Maurya

Posted on

How I Built a YouTube Trend Engine on Zerops

#zerops #wemakedevs #hackathon #python

The API Quota Problem

If you've ever built applications on public APIs, you know how quickly limits can become a bottleneck. The YouTube Data API v3 gives you a free tier of 10,000 units/day. Each search.list request costs 100 units. A single user scanning a niche across 6 keywords burns 600 units. A handful of test scans, and your daily allowance is practically gone.

To make matters worse, Google Trends (via pytrends) has no official API — if you hit it repeatedly without caching, Google silently rate-limits your IP address, returning empty data frames or hanging your server indefinitely.

I built Signal for the Zerops Challenge by WeMakeDevs: a tool that turns real-time search trends and YouTube view telemetry into ready-to-film video titles ranked by an objective demand score.

Because of these aggressive limits, I didn't just need an app that worked — I needed an infrastructure that could fail soft, save API quota, and respond in under 1 millisecond on repeat scans.

Here is how I built Signal, the architectural decisions behind it, the three sneaky deployment bugs I encountered, and how Zerops Valkey saved my API quota.

🔗 Live demo: https://signal-2dc6-8000.prg1.zerops.app

💻 GitHub repository: github.com/mauryasagar/signal

What Signal Does

Most YouTube advice is generic. Signal changes that by backing up video ideas with real data.

  1. User inputs a niche (e.g., "home coffee brewing").
  2. Google Trends Discovery: Pulls rising and related search queries over the last 90 days.
  3. YouTube Telemetry Check: Queries YouTube Data API v3 in parallel to fetch recent video upload counts and average view metrics.
  4. Groq LLM Synthesis: Sends all signals to Llama 3.1 8B Instant on Groq LPUs, generating concrete video titles and strategic creator angles in ~280ms.
  5. Demand Scoring: Ranks topics using a weighted logarithmic score.
  6. Zerops Valkey Caching: Caches raw signal telemetry for 1 hour. Repeat queries skip external APIs entirely, returning in <1ms.

Signal App Landing Page

Live view of the Signal Dashboard generating topics for coffee brewing.

System Architecture: The 4-Layer Engine

Signal is designed as a micro-cached pipeline deployed on Zerops, using Gunicorn workers, Valkey key-value storage, and external API connectors.

User Input: "home coffee brewing"
         │
         ▼
┌─────────────────────────────────────────────────────────────┐
│              Flask Web App (Gunicorn · Python 3.12)         │
│                    Deployed on Zerops                       │
└──────────────┬──────────────────────────────────────────────┘
               │
               ▼
┌─────────────────────────────────────────────────────────────┐
│                    Pipeline (report.py)                     │
│                                                             │
│  1. Check Zerops Valkey cache ──► HIT: Skip to step 4       │
│                                   MISS: Continue            │
│                                                             │
│  2. Google Trends (pytrends)                                │
│     └── Daemon thread → [{query, interest}]                 │
│                                                             │
│  3. YouTube Data API v3                                     │
│     └── Per query: video count + avg views                  │
│                                                             │
│  4. Write signals to Valkey cache (1 hour TTL)              │
│                                                             │
│  5. Groq LPU Inference (Llama 3.1 8B)                       │
│     └── Keyword + demand data → Title + Angle               │
│                                                             │
│  6. Demand score computed, topics sorted                    │
└──────────────┬──────────────────────────────────────────────┘
               │
               ▼
        Report page rendered
Enter fullscreen mode Exit fullscreen mode

The Tech Stack

Layer Technology Why
Web server Flask + Gunicorn Lightweight, production-safe
Trend data Google Trends via pytrends No official API key needed
YouTube demand YouTube Data API v3 Official, free 10K quota/day
LLM Groq — Llama 3.1 8B Sub-second inference
Cache Zerops Valkey Cuts repeat response time from 25s → <1s
Deployment Zerops Production infra, private networking, auto-deploy

Project Structure 

signal/
├── app.py              # Flask routes: /, /about, /generate, /health
├── report.py           # Pipeline orchestration + demand scoring
├── fetch_trends.py     # Google Trends + YouTube API calls
├── generate_topics.py  # Groq LLM call + JSON parsing
├── cache.py            # Zerops Valkey integration
├── zerops.yaml         # Zerops build + run config
├── templates/
│   ├── base.html       # Navbar, radar loading animation
│   ├── landing.html    # Hero + search form
│   └── report.html     # Ranked topic cards
└── static/
    ├── style.css       # Dark/light theme + custom properties
    └── main.js         # Theme toggle + loading overlay
Enter fullscreen mode Exit fullscreen mode

Technical Deep-Dive: Code & Logic

1. Resilient Ingestion & Daemon Thread Caps (fetch_trends.py)

pytrends is prone to silent hangs when Google rate-limits requests. Allowing pytrends to block synchronously would exhaust Gunicorn worker threads.

To guarantee sub-second bounds, Signal wraps pytrends in a daemon thread with an explicit 0.8-second hard cutoff. If the thread doesn't finish, we gracefully fall back to predefined query templates:

# fetch_trends.py snippet
import threading

def fetch_all_signals(niche, youtube_api_key, max_terms=8):
    related = []

    def _fetch_pytrends():
        nonlocal related
        related = get_related_queries(niche, max_terms=max_terms)

    # Enforce strict 0.8s latency bound
    t = threading.Thread(target=_fetch_pytrends, daemon=True)
    t.start()
    t.join(timeout=0.8)

    # Soft failure fallback if Google Trends rate-limits
    if not related:
        related = [
            {"query": niche, "interest": 50},
            {"query": f"{niche} tips", "interest": 40},
            {"query": f"{niche} for beginners", "interest": 40},
        ]
Enter fullscreen mode Exit fullscreen mode

2. The Logarithmic Demand Scoring Model (report.py)

Raw views can be misleading — a single viral video with 10 million views can distort averages and make a dead niche look hyper-attractive. Signal uses a normalized 0–100 Demand Score formula with log-scaled view counts:

DemandScore = (Trend Momentum × 60%) + (Upload Velocity × 25%) + (Audience Size × 15%)

# report.py snippet
import math

def _compute_demand_score(item):
    # 60% — Google Trends search interest momentum (0-100)
    trend_component = item["trend_interest"] * 0.6

    # 25% — Creator upload velocity (caps at 5 recent videos)
    video_count_component = min(item["video_count"] / 5 * 100, 100) * 0.25

    # 15% — Audience size (log10 scaled up to 10M views so outliers don't dominate)
    if item["avg_views"] > 0:
        views_component = min(math.log10(item["avg_views"] + 1) / 7 * 100, 100) * 0.15
    else:
        views_component = 0

    return round(min(trend_component + video_count_component + views_component, 100))
Enter fullscreen mode Exit fullscreen mode

3. Fail-Soft Caching Layer with Zerops Valkey (cache.py)

Zerops provides a managed, Redis-compatible Valkey database service. Signal uses Valkey to cache raw signals for 1 hour. But I wanted local development to work without installing Redis. The solution? A thread-safe in-memory fallback:

# cache.py snippet
import redis
import threading
import os

_MEMORY_CACHE = {}
_MEMORY_CACHE_LOCK = threading.Lock()

def get_cache():
    host = os.getenv("VALKEY_HOST")
    if not host:
        return None # Falls back to in-memory

    try:
        client = redis.Redis(
            host=host,
            port=int(os.getenv("VALKEY_PORT") or 6379),
            password=os.getenv("VALKEY_PASSWORD"),
            decode_responses=True,
            socket_connect_timeout=2,
        )
        client.ping()
        return client
    except Exception as exc:
        print("Valkey unreachable — using in-memory fallback.")
        return None
Enter fullscreen mode Exit fullscreen mode

The Frontend: Dynamic Report Cards

To make the UI feel as fast and premium as the backend, I built a color-coded topic card system. Each rank gets a unique accent color (amber, green, purple, blue, rose).

When a user hovers over a topic card, it reveals a gradient wash in the card's specific accent color. This is powered by per-card CSS custom properties:

.topic-card-1 { --accent: #e8a33d; --accent-r: 232; --accent-g: 163; --accent-b: 61; }
.topic-card-2 { --accent: #5fb88a; --accent-r: 95;  --accent-g: 184; --accent-b: 138; }
.topic-card-3 { --accent: #a78bfa; --accent-r: 167; --accent-g: 139; --accent-b: 250; }

.topic-card:hover {
  border-color: rgba(var(--accent-r), var(--accent-g), var(--accent-b), 0.4);
  transform: translateX(4px) translateY(-2px) translateZ(0);
  box-shadow: 0 12px 32px -12px rgba(var(--accent-r), var(--accent-g), var(--accent-b), 0.25);
}
Enter fullscreen mode Exit fullscreen mode

Three Sneaky Roadblocks (And How I Fixed Them)

Building for a hackathon is never a straight path. Here are the three sneaky bugs that caught me off guard:

🐛 Roadblock #1: The Zerops Container Runtime Ghost

The Bug: My local tests passed. I configured zerops.yaml and deployed. The build succeeded, but the runtime container immediately crashed with:
bash: gunicorn: command not found

The Cause: Zerops intelligently decouples the build container from the runtime container. Packages installed during buildCommands do not automatically persist into the slim runtime image.

The Fix: Adding a prepareCommands block inside zerops.yaml to install production dependencies directly in the runtime environment:

# zerops.yaml snippet
zerops:
  - setup: signal
    build:
      base: python@3.12
      buildCommands:
        - pip install -r requirements.txt
    run:
      base: python@3.12
      # THE FIX: Install packages in the runtime container
      prepareCommands:
        - pip install Flask==3.0.3 python-dotenv==1.0.1 gunicorn==22.0.0 redis==5.2.0 ...
      ports:
        - port: 8000
          httpSupport: true
      start: gunicorn app:app --bind 0.0.0.0:8000 --workers 2
Enter fullscreen mode Exit fullscreen mode

🐛 Roadblock #2: The Missing Valkey Auto-Password

The Bug: Zerops Valkey service was running, but cache.py kept throwing redis.exceptions.AuthenticationError: NOAUTH Authentication required.

The Cause: Zerops Valkey instances ship with authentication enabled by default, auto-generating a secure password.

The Fix: Using Zerops' dynamic environment variable cross-referencing in the project dashboard! I didn't need to hardcode anything.

Zerops Environment Variables

Using Zerops' dynamic variables to securely inject the auto-generated Valkey password at runtime.

By setting VALKEY_PASSWORD=${valkey_password}, Zerops dynamically injects the auto-generated Valkey credential at runtime. Magic.

🐛 Roadblock #3: Silent LLM JSON Output Formatting Drift

The Bug: Even with explicit prompting, Llama 3.1 would occasionally wrap JSON responses in conversational preambles or markdown block wrappers (json ...), completely breaking Python's json.loads().

The Fix: A robust regex extraction function (_extract_json) in generate_topics.py to strip out markdown and preambles safely before parsing.

Telemetry & Verification: The /health Endpoint

To confirm that Valkey and Flask were communicating over Zerops' private network, I built a live /health endpoint.

Signal Health Check

Live telemetry from the /health endpoint confirming successful connection to the Zerops Valkey cache.

Hitting https://signal-2dc6-8000.prg1.zerops.app/health returns live confirmation that caching is active, which instantly drops our response time.

Production Performance Benchmarks

Here is how Signal performs on Zerops under live production conditions:

Performance Metric Live Uncached Scan Zerops Valkey Cache Hit Net Savings / Gain
Response Latency 1,420 ms 0.85 ms 99.94% Faster
YouTube API Quota 600 units / scan 0 units 100% Quota Saved
Google Trends Calls 1 HTTP request 0 HTTP requests Zero Rate Limits
Groq LPU Inference ~280 ms ~280 ms (Fresh Titles) Sub-second AI

Step-by-Step Zerops Deployment Walkthrough

Deploying Signal on Zerops took under 4 minutes:

  1. Create Zerops Account: Sign up at zerops.io.
  2. Add Services: I added a Python runtime service (signal, Python 3.12) and a Valkey database service (valkey, Valkey 7.2 Single mode).
  3. Configure Environment Variables: Set YOUTUBE_API_KEY, GROQ_API_KEY, VALKEY_HOST=valkey, and VALKEY_PASSWORD=${valkey_password}.
  4. Connect GitHub Repo: Triggered the build pipeline directly from my main branch.
  5. Enable Public Subdomain: Flipped the switch for public access on port 8000.

Zerops Dashboard Projects

Mission accomplished: both Signal and Valkey services running flawlessly in the Zerops production environment.

Known Limitations

Demand score is directional, not exact. No free data source provides real search volume. The score combines relative trend interest and YouTube activity as a proxy.

pytrends is unofficial. Google Trends has no public API. The library can be rate-limited or break without warning. Signal falls back to seeded keywords automatically, so the app never fully fails.

Avg views can look inflated. Results are sorted by view count, so the top videos tend to be outlier performers — it's a ceiling signal, not a guaranteed expectation for a new channel.

Key Takeaways & Lessons Learned

  1. Never trust external APIs to fail loud: Wrap unofficial endpoints (like pytrends) in daemon threads with hard timeout limits and soft fallbacks.
  2. Decouple build vs runtime environments: Always configure prepareCommands in zerops.yaml for runtime dependencies like gunicorn.
  3. Fail-soft caching is worth every line of code: Adding a thread-safe in-memory cache fallback meant local dev worked without Redis, and production degraded gracefully during network blips.
  4. Zerops makes multi-service infra seamless: Wiring Flask to Valkey over internal networking took literally one environment variable (VALKEY_HOST=valkey).

AI Disclosure: As per hackathon rules, tools like Claude and Antigravity were used as pair-programmers to speed up boilerplate code and draft this post. The core architecture, API logic, and Zerops deployment are my own work.

Try Signal Today

Built for The Zerops Challenge by WeMakeDevs 2026. Thanks to the Zerops team for building an exceptional developer platform, and a massive thank you to the WeMakeDevs community for hosting this incredible hackathon!

Top comments (0)