How I Stopped One Viral Reel From Lying to My Analytics Pipeline

If you scrape competitor accounts to find content ideas, you'll hit a problem almost immediately: outliers lie.

One Reel that randomly goes viral doesn't mean the account is suddenly amazing at everything. But if you're not careful, that single spike becomes the new "baseline" your whole system judges every future post against — and you start missing genuinely good posts because they don't look impressive next to a fluke.

I ran into this while building Vcentre — a nightly competitor-intelligence pipeline that scrapes Instagram accounts, finds outlier posts, and turns them into creative briefs for my own content bots. Here's the architecture, the real production numbers, and the reasoning behind the gates that keep one lucky Reel from corrupting the entire dataset.

1. The Real Math Proof: Reels vs. Photos (The 26x Gap)

This isn't a theoretical problem. Here's the actual distribution gap sitting in my production database, across 254 competitor posts (149 photos/carousels, 105 reels) indexed from 18 target accounts:

Metric Photos / Carousels Video Reels Difference
Average Engagement 1,966 likes 51,271 views 26x higher
Max Peak Outlier 68,831 likes 1,309,461 views 19x higher

If Reels and Photos were pooled into one dataset, a single 1.3M-view Reel would poison the median for the entire account — and every genuinely high-performing photo would look completely dead by comparison.

This is exactly why Vcentre scores Reels and Photos as separate cohorts, each with its own median and its own outlier threshold. A viral Reel can only skew the baseline for other Reels — it has zero effect on how photos get judged.

python
def compute_cohort_baselines(posts: list[dict]) -> dict:
cohorts = {"reel": [], "photo": []}
for post in posts:
cohorts[post["media_type"]].append(post["engagement_rate"])

return {
    media_type: {
        "median": statistics.median(rates) if rates else 0,
        "sample_size": len(rates)
    }
    for media_type, rates in cohorts.items()
}

Two real outliers the engine actually caught with this approach: one account broke out to over a million views on a career/salary-hook Reel — more than 20x above its own baseline. Another cleared a huge comment count on a Claude Code vs. ChatGPT comparison Reel. Neither would've registered as anomalous if judged against a mixed-format baseline instead of its own cohort.

1. The Engagement Formula: Comments Aren't Weighted Like Likes

Raw likes / views isn't enough — it treats a low-friction like the same as a high-intent comment. Vcentre's engagement formula weights comments noticeably heavier than likes:

python
def engagement_rate(post: dict) -> float:
return (post["likes"] + (post["comments"] * COMMENT_WEIGHT)) / post["views"]

Comments require someone to stop, think, and type — they're a much stronger virality signal than a passive like. The exact weight took some tuning to get right, but the principle matters more than the number: weighting comments heavier filters out posts that just got a view-spike from the algorithm without any real audience reaction.

1. The Three-Gate Threshold

Once a post has a cohort-relative baseline to compare against, it still has to clear three independent gates before it's worth spending LLM budget on. Any one signal alone is too noisy:

A relative-outlier threshold — how far above its cohort's median a post needs to be
An absolute floor — protects against tiny accounts where a "big relative jump" off a near-zero baseline is meaningless
A minimum engagement rate — filters out posts that only look good because of a tiny follower count
python
def is_worth_analyzing(post: dict, cohort_baseline: dict) -> bool:
median = cohort_baseline["median"]

clears_relative = post["engagement_rate"] >= median * RELATIVE_MULTIPLIER
clears_floor = post["engagement_rate"] >= ABSOLUTE_FLOOR
clears_min_rate = post["engagement_rate"] >= MIN_ENGAGEMENT_RATE

return clears_relative and clears_floor and clears_min_rate

The exact constants are tuned to my specific accounts and niche — what matters architecturally is that no single signal is trusted alone.

1. The Maturation Guard and Recency Decay

Two more details that matter more than they look:

A maturation floor. Instagram takes roughly a day to distribute a Reel beyond an account's existing followers. Scoring a post before that window closes produces false positives and false negatives, so Vcentre refuses to touch anything too fresh, and also ignores anything too old so stale topics don't compete with today's queue.

A recency decay curve. A big Reel from three weeks ago shouldn't outrank an equally big Reel from two days ago just because it happened to be scraped in the same batch. Vcentre applies a decay multiplier that fades a post's score the older it gets, so today's outliers always get priority over yesterday's news.

1. Guardrails: Cooldowns and Circuit Breakers

Scraping Instagram at scale is a great way to get rate-limited or banned if you're not careful. Every scrape target is guarded by a cooldown window per account, plus a circuit breaker that trips after repeated failures and stops hitting that account entirely until it resets.

python
class ScrapeCircuitBreaker:
def init(self, failure_threshold: int, cooldown_hours: int):
self.failure_threshold = failure_threshold
self.cooldown_hours = cooldown_hours
self.failures: dict[str, int] = {}
self.tripped_until: dict[str, datetime] = {}

def can_scrape(self, account: str) -> bool:
    trip_time = self.tripped_until.get(account)
    if trip_time and datetime.utcnow() < trip_time:
        return False
    return True

def record_failure(self, account: str):
    self.failures[account] = self.failures.get(account, 0) + 1
    if self.failures[account] >= self.failure_threshold:
        self.tripped_until[account] = datetime.utcnow() + timedelta(hours=self.cooldown_hours)
        print(f"Circuit breaker tripped for {account}. Cooling down.")

def record_success(self, account: str):
    self.failures[account] = 0

1. The 10-Provider Fallback Chain (and Why It's Free)

Every post that clears the gates above gets analyzed — but not by one expensive model. Groq and Cerebras handle the bulk of the analysis for free, splitting load across ten total providers. Gemini is only invoked once, at the very end, to synthesize the final brief from everything the free models already extracted. Before that synthesis happens, the pipeline also fuses in live signals from Google Trends, Hacker News' top headlines, and a tech-news feed — so the brief isn't just analyzing a competitor in a vacuum, it's grounding the hook in whatever's breaking today.

python
ANALYSIS_PROVIDERS = [
("groq", "llama-3.3-70b-versatile"),
("cerebras", "gpt-oss-120b"),
("openrouter", "gemma-3-1b"),
# ...7 more fallback providers
]

def analyze_outlier(post: dict) -> dict:
for provider, model in ANALYSIS_PROVIDERS:
try:
return run_analysis(provider, model, post)
except ProviderError:
continue
raise AllProvidersFailedError(post["id"])

def synthesize_brief(analyses: list[dict], trend_signals: dict) -> dict:
# Only Gemini touches this step — everything above it was free
return gemini_client.synthesize(analyses, context=trend_signals)

This gating — cheap models do the volume work, the paid model only does final synthesis — is the same trick that got Veltrix's cost per post down to fractions of a cent. It works just as well here: 32 structured creative briefs generated so far, at $0 in monthly infra cost.

1. Closing the Loop

The briefs Vcentre generates don't just sit in a report — they write straight into the same database my posting bots read from. And a feedback job scores each published post's real performance back against the pattern that flagged it:

If a published post clearly outperforms baseline, that pattern's score gets boosted
If it clearly underperforms, the pattern's score is penalized
If a post barely got any distribution at all, the system refuses to penalize the pattern — protecting against flukes that aren't the pattern's fault
Patterns that get used repeatedly and keep scoring poorly are automatically retired
python
def score_feedback_loop(published_post: dict, source_brief: dict):
actual_engagement = fetch_current_engagement(published_post["id"])
predicted_engagement = source_brief["expected_engagement"]

accuracy = 1 - abs(actual_engagement - predicted_engagement) / predicted_engagement
log_pattern_accuracy(source_brief["pattern_id"], accuracy)

Key Metrics From Real Production Runs
254 competitor posts indexed across 18 target accounts (149 photos, 105 reels)
26x average volume gap between Reels and Photos — successfully isolated by cohort scoring
Top outlier caught: north of a million views on a single breakout Reel
32 structured creative briefs synthesized directly into the publishing queue
Monthly infra/API cost: $0.00

Cohort-aware baselines sound like a small detail, but they're the difference between a system that chases noise and one that actually finds signal. If your pipeline treats every post as coming from the same distribution, you're probably reacting to outliers you shouldn't be.

Let me know how you handle cohort skew or outlier detection in your own scraping pipelines!

Check out the full interactive workspace at theayush.pages.dev.

Comments

[Ayush Agarwal]

Hey dev community! 👋 I'm Ayush, a 17-year-old CSE student at VIT Vellore and the creator of Vcentre — a competitor-intelligence pipeline I built to turn viral outliers into creative briefs without getting fooled by a single lucky Reel skewing the whole dataset.

Vcentre is one of three production systems I shipped this year, alongside Vurlo (e-commerce SaaS) and Veltrix (the autonomous social engine I posted about earlier). I built all three to learn how to scale real systems on close to zero budget.

I'd love to hear your feedback on the cohort-scoring approach, or how you handle outlier detection in your own scraping/analytics pipelines.

You can check out all my active projects and their codebases on my portfolio: theayush.pages.dev

Let me know if you have any questions! 🚀