DEV Community: Ayush Agarwal

How I Stopped One Viral Reel From Lying to My Analytics Pipeline

Ayush Agarwal — Wed, 26 Aug 2026 08:48:21 +0000

If you scrape competitor accounts to find content ideas, you'll hit a problem almost immediately: outliers lie.

One Reel that randomly goes viral doesn't mean the account is suddenly amazing at everything. But if you're not careful, that single spike becomes the new "baseline" your whole system judges every future post against — and you start missing genuinely good posts because they don't look impressive next to a fluke.

I ran into this while building Vcentre — a nightly competitor-intelligence pipeline that scrapes Instagram accounts, finds outlier posts, and turns them into creative briefs for my own content bots. Here's the architecture, the real production numbers, and the reasoning behind the gates that keep one lucky Reel from corrupting the entire dataset.

The Real Math Proof: Reels vs. Photos (The 26x Gap)

This isn't a theoretical problem. Here's the actual distribution gap sitting in my production database, across 254 competitor posts (149 photos/carousels, 105 reels) indexed from 18 target accounts:

Metric Photos / Carousels Video Reels Difference
Average Engagement 1,966 likes 51,271 views 26x higher
Max Peak Outlier 68,831 likes 1,309,461 views 19x higher

If Reels and Photos were pooled into one dataset, a single 1.3M-view Reel would poison the median for the entire account — and every genuinely high-performing photo would look completely dead by comparison.

This is exactly why Vcentre scores Reels and Photos as separate cohorts, each with its own median and its own outlier threshold. A viral Reel can only skew the baseline for other Reels — it has zero effect on how photos get judged.

python
def compute_cohort_baselines(posts: list[dict]) -> dict:
cohorts = {"reel": [], "photo": []}
for post in posts:
cohorts[post["media_type"]].append(post["engagement_rate"])

return {
    media_type: {
        "median": statistics.median(rates) if rates else 0,
        "sample_size": len(rates)
    }
    for media_type, rates in cohorts.items()
}

Two real outliers the engine actually caught with this approach: one account broke out to over a million views on a career/salary-hook Reel — more than 20x above its own baseline. Another cleared a huge comment count on a Claude Code vs. ChatGPT comparison Reel. Neither would've registered as anomalous if judged against a mixed-format baseline instead of its own cohort.

The Engagement Formula: Comments Aren't Weighted Like Likes

Raw likes / views isn't enough — it treats a low-friction like the same as a high-intent comment. Vcentre's engagement formula weights comments noticeably heavier than likes:

python
def engagement_rate(post: dict) -> float:
return (post["likes"] + (post["comments"] * COMMENT_WEIGHT)) / post["views"]

Comments require someone to stop, think, and type — they're a much stronger virality signal than a passive like. The exact weight took some tuning to get right, but the principle matters more than the number: weighting comments heavier filters out posts that just got a view-spike from the algorithm without any real audience reaction.

The Three-Gate Threshold

Once a post has a cohort-relative baseline to compare against, it still has to clear three independent gates before it's worth spending LLM budget on. Any one signal alone is too noisy:

A relative-outlier threshold — how far above its cohort's median a post needs to be
An absolute floor — protects against tiny accounts where a "big relative jump" off a near-zero baseline is meaningless
A minimum engagement rate — filters out posts that only look good because of a tiny follower count
python
def is_worth_analyzing(post: dict, cohort_baseline: dict) -> bool:
median = cohort_baseline["median"]

clears_relative = post["engagement_rate"] >= median * RELATIVE_MULTIPLIER
clears_floor = post["engagement_rate"] >= ABSOLUTE_FLOOR
clears_min_rate = post["engagement_rate"] >= MIN_ENGAGEMENT_RATE

return clears_relative and clears_floor and clears_min_rate

The exact constants are tuned to my specific accounts and niche — what matters architecturally is that no single signal is trusted alone.

The Maturation Guard and Recency Decay

Two more details that matter more than they look:

A maturation floor. Instagram takes roughly a day to distribute a Reel beyond an account's existing followers. Scoring a post before that window closes produces false positives and false negatives, so Vcentre refuses to touch anything too fresh, and also ignores anything too old so stale topics don't compete with today's queue.

A recency decay curve. A big Reel from three weeks ago shouldn't outrank an equally big Reel from two days ago just because it happened to be scraped in the same batch. Vcentre applies a decay multiplier that fades a post's score the older it gets, so today's outliers always get priority over yesterday's news.

Guardrails: Cooldowns and Circuit Breakers

Scraping Instagram at scale is a great way to get rate-limited or banned if you're not careful. Every scrape target is guarded by a cooldown window per account, plus a circuit breaker that trips after repeated failures and stops hitting that account entirely until it resets.

python
class ScrapeCircuitBreaker:
def init(self, failure_threshold: int, cooldown_hours: int):
self.failure_threshold = failure_threshold
self.cooldown_hours = cooldown_hours
self.failures: dict[str, int] = {}
self.tripped_until: dict[str, datetime] = {}

def can_scrape(self, account: str) -> bool:
    trip_time = self.tripped_until.get(account)
    if trip_time and datetime.utcnow() < trip_time:
        return False
    return True

def record_failure(self, account: str):
    self.failures[account] = self.failures.get(account, 0) + 1
    if self.failures[account] >= self.failure_threshold:
        self.tripped_until[account] = datetime.utcnow() + timedelta(hours=self.cooldown_hours)
        print(f"Circuit breaker tripped for {account}. Cooling down.")

def record_success(self, account: str):
    self.failures[account] = 0

The 10-Provider Fallback Chain (and Why It's Free)

Every post that clears the gates above gets analyzed — but not by one expensive model. Groq and Cerebras handle the bulk of the analysis for free, splitting load across ten total providers. Gemini is only invoked once, at the very end, to synthesize the final brief from everything the free models already extracted. Before that synthesis happens, the pipeline also fuses in live signals from Google Trends, Hacker News' top headlines, and a tech-news feed — so the brief isn't just analyzing a competitor in a vacuum, it's grounding the hook in whatever's breaking today.

python
ANALYSIS_PROVIDERS = [
("groq", "llama-3.3-70b-versatile"),
("cerebras", "gpt-oss-120b"),
("openrouter", "gemma-3-1b"),
# ...7 more fallback providers
]

def analyze_outlier(post: dict) -> dict:
for provider, model in ANALYSIS_PROVIDERS:
try:
return run_analysis(provider, model, post)
except ProviderError:
continue
raise AllProvidersFailedError(post["id"])

def synthesize_brief(analyses: list[dict], trend_signals: dict) -> dict:
# Only Gemini touches this step — everything above it was free
return gemini_client.synthesize(analyses, context=trend_signals)

This gating — cheap models do the volume work, the paid model only does final synthesis — is the same trick that got Veltrix's cost per post down to fractions of a cent. It works just as well here: 32 structured creative briefs generated so far, at $0 in monthly infra cost.

Closing the Loop

The briefs Vcentre generates don't just sit in a report — they write straight into the same database my posting bots read from. And a feedback job scores each published post's real performance back against the pattern that flagged it:

If a published post clearly outperforms baseline, that pattern's score gets boosted
If it clearly underperforms, the pattern's score is penalized
If a post barely got any distribution at all, the system refuses to penalize the pattern — protecting against flukes that aren't the pattern's fault
Patterns that get used repeatedly and keep scoring poorly are automatically retired
python
def score_feedback_loop(published_post: dict, source_brief: dict):
actual_engagement = fetch_current_engagement(published_post["id"])
predicted_engagement = source_brief["expected_engagement"]

accuracy = 1 - abs(actual_engagement - predicted_engagement) / predicted_engagement
log_pattern_accuracy(source_brief["pattern_id"], accuracy)

Key Metrics From Real Production Runs
254 competitor posts indexed across 18 target accounts (149 photos, 105 reels)
26x average volume gap between Reels and Photos — successfully isolated by cohort scoring
Top outlier caught: north of a million views on a single breakout Reel
32 structured creative briefs synthesized directly into the publishing queue
Monthly infra/API cost: $0.00

Cohort-aware baselines sound like a small detail, but they're the difference between a system that chases noise and one that actually finds signal. If your pipeline treats every post as coming from the same distribution, you're probably reacting to outliers you shouldn't be.

Let me know how you handle cohort skew or outlier detection in your own scraping pipelines!

Check out the full interactive workspace at theayush.pages.dev.

How I Built Veltrix: An Autonomous Social Engine Orchestrating 18+ APIs on a $0.0002 Budget

Ayush Agarwal — Thu, 16 Jul 2026 12:22:04 +0000

Running a social media account solo is exhausting. You have to research niches, write captions, design slides, and publish twice a day, every day.
If you try to automate this using a single LLM (like GPT-4o) running on a simple cron job, you run into three massive walls:

Halucination & Bad Output: A single model will eventually write cringey hashtags or break the layout.
Ephemeral Environments: Running on a free hosting platform or GitHub Actions means your server is destroyed every run—saving state is incredibly difficult.
API Cost Runaways: Querying high-end creative models for every draft will drain your bank account. To solve this, I built Veltrix — an autonomous publishing engine that orchestrates 18+ active APIs and publishes twice daily to Instagram and Threads for roughly $0.0002 per post. Here is the exact architecture, the fallback gates, and the code. --- ## 1. The 18-API Key Topology Veltrix doesn't just call one model; it manages an entire footprint of APIs to handle data gathering, generation, verification, and publishing:
Core Generation & Audits: Gemini Pro (Primary Brain), Groq (Llama 70B), Cerebras (gpt-oss-120b), OpenRouter (Gemma 31B).
Media & Brand Graphics: SiliconFlow (Flux/SD3), Hugging Face Inference, Unsplash Image Search, Logo.dev, Brandfetch.
Database & Ops: Supabase, local SQLite (veltrix.db), Cloudinary (media hosting), Discord webhooks.
Direct Publishing & Git: Instagram Graph API, Threads API, GitHub Actions API. To protect my wallet, I set up a gated paid API model: Gemini Pro is only invoked for final generation after the draft successfully passes a series of free/low-cost verification audits. --- ## 2. The Architectural Workflow Here is how a post moves from a raw trigger to a published carousel slide:
GitHub Actions Cron Tick triggers the script.
Load Checkpoint JSON fetches any pending reviews.
Gemini drafts the topic and caption.
Adaptive Embedding Deduplication runs a similarity check.
Text Auditor (Groq & Cerebras) runs a dual-model consensus check.
Gemini generates the final creative slide data (only if audits pass).
Playwright Chromium renders and screenshots the HTML slide template.
Cloudinary hosts the images publicly.

9. Meta Graph API publishes the post to Instagram & Threads.

3. Core Code Implementations

Here are the key Python modules that handle the heavy lifting:

`text_auditor.py` — Adversarial Model Auditing

To prevent mistakes, I never let a model grade its own homework. This module queries Groq and Cerebras independently and requires a 2-out-of-2 consensus approval before opening the paid API gate:

import os
import json
from groq import Groq
from cerebras.client import Cerebras
# Init clients using our credentials config
groq_client = Groq(api_key=os.getenv("GROQ_API_KEY"))
cerebras_client = Cerebras(api_key=os.getenv("CEREBRAS_API_KEY"))
def audit_draft_with_llm(client, model_name: str, draft: str) -> dict:
    prompt = """
    Analyze this social media draft. You are an editor. Check for:
    1. Grammatical errors or cringey hashtag stuffing.
    2. Factuality and structural formatting.
    Return ONLY a raw JSON object: {"approved": true/false, "reason": "string"}
    """

    response = client.chat.completions.create(
        model=model_name,
        messages=[
            {"role": "system", "content": prompt},
            {"role": "user", "content": draft}
        ],
        response_format={"type": "json_object"}
    )
    return json.loads(response.choices[0].message.content)
def verify_text_content(draft: str) -> bool:
    # Heuristics pre-screen (costs $0)
    if len(draft.split("#")) > 5:
        print("Pre-screen failed: Emoji or hashtag density too high.")
        return False

    # Consensus chain run in parallel
    try:
        groq_result = audit_draft_with_llm(groq_client, "llama-3.3-70b-versatile", draft)
        cerebras_result = audit_draft_with_llm(cerebras_client, "gpt-oss-120b", draft)

        # Absolute consensus required (2 of 2)
        consensus = groq_result["approved"] and cerebras_result["approved"]
        if not consensus:
            print(f"Audit rejected. Groq: {groq_result['reason']} | Cerebras: {cerebras_result['reason']}")

        return consensus
    except Exception as e:
        # Fall back to OpenRouter free models if Groq/Cerebras fail
        print(f"Primary auditors failed: {e}. Routing to OpenRouter fallback...")
        return fallback_audit_openrouter(draft)
database.py — Making State Survive Container Destruction
Because GitHub Actions containers are completely destroyed after a job ends, I built a checkpoint serializer that stores base64-encoded media bytes locally. The next cron runner picks it up to verify and post it:

python


import json
import base64
from datetime import datetime, timedelta
CHECKPOINT_FILE = "pipeline_checkpoint.json"
def save_checkpoint(topic: str, caption: str, media_paths: list):
    payload = {
        "timestamp": datetime.utcnow().isoformat(),
        "status": "pending_review",
        "topic": topic,
        "caption": caption,
        "media": []
    }

    # Base64 encode local slide images to serialize into JSON
    for path in media_paths:
        with open(path, "rb") as image_file:
            encoded_bytes = base64.b64encode(image_file.read()).decode('utf-8')
            payload["media"].append({"path": path, "bytes": encoded_bytes})

    with open(CHECKPOINT_FILE, "w") as f:
        json.dump(payload, f)
    print("Checkpoint saved. Ready for human review window.")
def load_checkpoint() -> dict:
    try:
        with open(CHECKPOINT_FILE, "r") as f:
            data = json.load(f)

        # Self-expire checkpoints older than 24 hours to avoid stale runs
        created = datetime.fromisoformat(data["timestamp"])
        if datetime.utcnow() - created > timedelta(hours=24):
            print("Checkpoint expired. Aborting.")
            return {}

        return data
    except FileNotFoundError:
                return {}

Operational Fallback Logic When running 18 separate APIs, something will break. Veltrix handles this using multi-tiered fallbacks:

Database Fallback: Remote logs write to Supabase via webhooks. If Supabase is down, the system writes to local SQLite (veltrix.db) and commits a JSON update back to the Git repo.
Visual Slide Generation: Premium slide illustrations are generated via SiliconFlow (Flux). If SiliconFlow rate-limits, it drops back to Hugging Face, and finally falls back to Gemini's internal image creator.
Brand Assets: In carousels comparing tools, logos are resolved by Logo.dev ➡️ Brandfetch API ➡️ Brandfetch CDN ➡️ Google Favicon API ➡️ local text fallback.
Key Metrics Since Launch
Inference cost / post: ~$0.0002 (Thanks to gating Gemini behind Llama)
API limits hit: 0 (Thanks to 12h handle cooldowns and circuit breakers)
Silent crashes: 0 (Thanks to health checks monitoring budget-aware failures vs quiet days)
By treating models as independent validating agents rather than single sources of truth, you can deploy fully autonomous cron-bots with zero risk of public failure on virtually zero budget.

Let me know how you guys handle quotas and secrets in your pipelines!