DEV Community: Olamide Olaniyan

How to Build a Tool that Track Which World Cup Players Are Blowing Up on Social Media

Olamide Olaniyan — Tue, 23 Jun 2026 09:56:00 +0000

Every World Cup there's a moment. Some player nobody outside their domestic league had heard of scores an absolute screamer in a knockout match, and by the time they've finished celebrating, their follower count is climbing like a rocket.

I always found that fascinating, but I could never see it happening in real time. By the time the "X gained 3 million followers!" tweets show up, the surge is already over.

So this tournament I built a little tracker that snapshots player follower counts on a schedule and shows me the growth curve as it happens. Here's how it works.

Why the official APIs were a dead end

My first instinct was to do this "properly" with official APIs. That died fast:

Instagram's Graph API won't give you follower counts for accounts you don't own.
TikTok's research API is academics-only and takes weeks of applications.
X's API now starts at $100/month.

I just wanted public follower counts — numbers anyone can see by opening the app. I didn't want a data partnership and a legal review.

I ended up using SociaVault, which wraps the public profile data from each platform behind one API and one key. One request, one credit, JSON back.

The shared client

Everything runs through one tiny helper:

// Node 18+ has fetch built in
const API_KEY = process.env.SOCIAVAULT_API_KEY;
const BASE = "https://api.sociavault.com";

async function sv(path, params) {
  const url = new URL(BASE + path);
  Object.entries(params).forEach(([k, v]) => url.searchParams.set(k, v));
  const res = await fetch(url, { headers: { "X-API-Key": API_KEY } });
  if (!res.ok) throw new Error(`${res.status} ${await res.text()}`);
  return res.json();
}

Grabbing follower counts across platforms

Each platform nests the count slightly differently, so I use fallback chains to stay defensive:

async function instagramFollowers(username) {
  const data = await sv("/v1/scrape/instagram/profile", { username });
  const p = data.data?.user ?? data.data ?? data;
  return p.follower_count ?? p.edge_followed_by?.count ?? null;
}

async function tiktokFollowers(username) {
  const data = await sv("/v1/scrape/tiktok/profile", { username });
  const stats = data.stats ?? data.user?.stats ?? {};
  return stats.followerCount ?? null;
}

async function xFollowers(username) {
  const data = await sv("/v1/scrape/twitter/profile", { username });
  const u = data.user ?? data.data ?? data;
  return u.followers_count ?? null;
}

Snapshot a whole watchlist at once

I keep a list of players and their handles, then snapshot everyone in one pass. Promise.allSettled means one bad lookup never sinks the whole run:

const watchlist = [
  { name: "Breakout Striker", ig: "player_ig", tiktok: "player_tt", x: "player_x" },
  // add as many as you want
];

async function snapshot() {
  const ts = new Date().toISOString();
  const results = await Promise.allSettled(
    watchlist.map(async (p) => ({
      name: p.name,
      ts,
      ig: await instagramFollowers(p.ig),
      tiktok: await tiktokFollowers(p.tiktok),
      x: await xFollowers(p.x),
    }))
  );
  return results.filter(r => r.status === "fulfilled").map(r => r.value);
}

Append each run to a CSV (or a real DB), run it on a cron every hour during the tournament, and within a couple of matches you've got a clean time series.

The fun part: spotting the breakout

The story lives in the percentage growth, not the absolute numbers. A superstar gaining 500k followers is a smaller relative event than an unknown gaining 500k off a base of 200k. Sort your deltas by percentage growth and the breakout players float right to the top — usually before the mainstream "look who blew up" posts even start.

Cost

Each profile lookup is one credit. A watchlist of 20 players across 3 platforms, snapshotted hourly for a month, is a rounding error compared to enterprise social tools.

Want the full version?

I wrote up the complete build — CSV logging, surge-detection math, charting — on the SociaVault blog: Tracking Player Social Growth During the World Cup. There's also a more story-driven piece on why these follower surges happen.

Grab a free key at sociavault.com — you get 50 credits, which is plenty to pilot a watchlist.

What would you point this at next? I'm tempted to try a full F1 season.

I Built a Tool to Track Which World Cup Players Are Blowing Up on Social Media

Olamide Olaniyan — Mon, 22 Jun 2026 09:56:00 +0000

I always found that fascinating, but I could never see it happening. By the time the "X gained 3M followers!" tweets show up, the surge is already over. So this tournament I built a little tracker that snapshots player follower counts on a schedule and shows me the growth curve in near real-time.

Here's how it works.

The problem with doing this "properly"

My first instinct was the official APIs. That died fast.

Instagram's Graph API won't give you follower counts for accounts you don't own.
TikTok's Research API is academics-only and takes weeks of applications.
X's API now starts at $100/month and climbs steeply from there.

I just wanted public follower counts — numbers anyone can see by opening the app. I didn't want a data partnership and a legal review.

I ended up using the SociaVault API, which wraps public profile data from each platform behind one key. One request, one credit, JSON back.

The shared client

Everything runs through one tiny helper:

// Node 18+ has fetch built in
const API_KEY = process.env.SOCIAVAULT_API_KEY;
const BASE = "https://api.sociavault.com";

async function sv(path, params) {
  const url = new URL(BASE + path);
  Object.entries(params).forEach(([k, v]) => url.searchParams.set(k, v));
  const res = await fetch(url, { headers: { "X-API-Key": API_KEY } });
  if (!res.ok) throw new Error(`${res.status} ${await res.text()}`);
  return res.json();
}

Grabbing follower counts across platforms

Each platform nests the count slightly differently, so I use fallback chains to stay defensive:

async function instagramFollowers(username) {
  const data = await sv("/v1/scrape/instagram/profile", { username });
  const p = data.data?.user ?? data.data ?? data;
  return p.follower_count ?? p.edge_followed_by?.count ?? null;
}

async function tiktokFollowers(username) {
  const data = await sv("/v1/scrape/tiktok/profile", { username });
  const stats = data.stats ?? data.user?.stats ?? {};
  return stats.followerCount ?? null;
}

async function xFollowers(username) {
  const data = await sv("/v1/scrape/twitter/profile", { username });
  const u = data.user ?? data.data ?? data;
  return u.followers_count ?? null;
}

Snapshot a whole watchlist at once

I keep a list of players and their handles, then snapshot everyone in one pass. Promise.allSettled means one bad lookup never sinks the whole run:

const watchlist = [
  { name: "Breakout Striker", ig: "player_ig", tiktok: "player_tt", x: "player_x" },
  // add as many as you want
];

async function snapshot() {
  const ts = new Date().toISOString();
  const results = await Promise.allSettled(
    watchlist.map(async (p) => ({
      name: p.name,
      ts,
      ig: await instagramFollowers(p.ig),
      tiktok: await tiktokFollowers(p.tiktok),
      x: await xFollowers(p.x),
    }))
  );
  return results.filter(r => r.status === "fulfilled").map(r => r.value);
}

Append each run to a CSV (or a real DB), run it on a cron every hour during the tournament, and within a couple of matches you've got a clean time series.

The fun part: spotting the breakout

The story lives in the percentage growth, not the absolute numbers. A superstar gaining 500k followers is a smaller relative event than an unknown gaining 500k off a base of 200k. Sort your deltas by percentage growth and the breakout players float right to the top — usually before the mainstream "look who blew up" posts even start.

Cost

Each profile lookup is one credit. A watchlist of 20 players across 3 platforms, snapshotted hourly for a month, is cheap — a rounding error next to what enterprise social tools charge.

Want the full version?

Grab a free key at sociavault.com — you get 50 credits, plenty to pilot a watchlist.

What would you point this at? I'm tempted to run the same setup on a Formula 1 season next.

I Built a Match-Day Social Listening Dashboard for the World Cup in a Weekend

Olamide Olaniyan — Fri, 19 Jun 2026 09:54:00 +0000

I wanted a single screen that told me what the internet was feeling during a World Cup match — which teams and players were spiking, whether the mood was joy or rage, and which moments were about to go viral. The enterprise social listening tools that do this cost more per month than my rent. So I built my own over a weekend.

Here's the architecture and the code that matters.

What I wanted on the screen

Live conversation volume per team/player (who's spiking right now)
Sentiment per platform (X furious, TikTok celebrating — they're rarely the same)
A simple "something's happening" alert when volume jumps
Refresh every couple of minutes during the match

The data layer

I didn't want to wrangle five different official APIs (and get rate-limited or rejected by most of them). I used the SociaVault API so every platform speaks the same shape behind one key.

A single fetch helper covers everything:

const API_KEY = process.env.SOCIAVAULT_API_KEY;
const BASE = "https://api.sociavault.com";

async function sv(path, params) {
  const url = new URL(BASE + path);
  Object.entries(params).forEach(([k, v]) => url.searchParams.set(k, v));
  const res = await fetch(url, { headers: { "X-API-Key": API_KEY } });
  if (!res.ok) throw new Error(`${res.status}`);
  return res.json();
}

Pulling the conversation for a topic

For a given match or player, I pull recent posts/comments from a few platforms and normalize them to plain text strings. The sentiment step doesn't care which platform a string came from:

async function gatherConversation(query) {
  const [x, tiktok] = await Promise.all([
    sv("/v1/scrape/twitter/search", { query, limit: "100" }),
    sv("/v1/scrape/tiktok/search/keyword", { query, limit: "100" }),
  ]);

  return {
    x: (x.tweets ?? x.data ?? []).map(t => t.text ?? ""),
    tiktok: (tiktok.data ?? []).map(v => v.desc ?? ""),
  };
}

A transparent sentiment score

I started with a tiny lexicon scorer. It's crude, but it's transparent and fast, and you can tune it for football slang (in match conversation, "penalty" and "VAR" usually skew negative because they show up when fans are furious):

const POSITIVE = new Set(["amazing", "goal", "wins", "class", "magic", "hero"]);
const NEGATIVE = new Set(["var", "penalty", "robbed", "disgrace", "offside", "bottled"]);

function scoreCorpus(texts) {
  let pos = 0, neg = 0;
  for (const t of texts) {
    for (const w of t.toLowerCase().split(/\W+/)) {
      if (POSITIVE.has(w)) pos++;
      if (NEGATIVE.has(w)) neg++;
    }
  }
  const total = pos + neg || 1;
  return { score: (pos - neg) / total, pos, neg, n: texts.length };
}

Run it per platform and the spread is the insight. During a contentious match you'll often see TikTok positive (highlights), X negative (refereeing rage), Reddit in the middle. A single blended number would hide all of that.

The "something's happening" alert

Volume is the earliest signal. Store a rolling baseline, and when the latest window jumps well above it, fire an alert:

function isSpike(current, baseline, factor = 2.5) {
  return baseline > 0 && current >= baseline * factor;
}

That's the whole trick behind "catching a moment early" — you're watching the derivative, not the absolute count.

The front end

Honestly, the front end is the easy part. I threw the normalized numbers into a basic React dashboard with a few line charts (volume over time per topic, sentiment over time per platform) and a list of active alerts. Poll the backend every 90 seconds during a match and you've got a live cockpit.

Cost reality check

Each search call is one credit. Polling a handful of topics every 90 minutes... sorry, every 90 seconds, during a 2-hour match adds up, so I cache aggressively and diff results so I never reprocess content I've already seen. Still came out to a few dollars for a full match day.

The full write-up

I documented the complete dashboard build — backend, caching, charts, the whole thing — here: Build a World Cup Social Listening Dashboard. The real-time monitoring foundation is in Tracking World Cup Buzz in Real Time, and the sentiment deep-dive is here.

Free key with 50 credits at sociavault.com if you want to build your own. Genuinely one of the more fun weekend projects I've done — the moment your sentiment line tanks two seconds after a VAR decision, you feel like you've tapped into the planet's mood.

I Replaced 5 Social Media APIs With One Key (and My Code Got Way Simpler)

Olamide Olaniyan — Thu, 18 Jun 2026 09:53:00 +0000

A while back I was building a side project that needed public data from a few social platforms. Nothing crazy — profiles, posts, some engagement numbers. I figured I'd just grab each platform's official API.

Reader, I did not "just grab each platform's official API."

Here's what that road actually looked like, and how I ended up consolidating everything down to one key and roughly ten lines of shared code.

The five-API nightmare

Instagram (Meta Graph API). Great if you own the account. Useless for pulling public data about accounts you don't. Endless app review.

TikTok. The research API is academics-only with a long application. For commercial use, basically nothing.

X (Twitter). Used to be wonderful. Now $100/month to start, more for anything serious.

YouTube. Honestly the best of the bunch — generous and well-documented. Credit where due.

LinkedIn. Partner-only. For most people, no useful public access at all.

So to cover five platforms I was looking at: five sets of credentials, five auth flows, five rate-limit models, five totally different response shapes, two flat-out rejections, and a monthly bill. For a side project.

What I actually wanted

getProfile("tiktok", "someuser")
getProfile("instagram", "someuser")
getProfile("twitter", "someuser")

Same call shape, same auth, same error handling. That's it. I don't care that each platform structures things differently internally — I want one boundary that hides that from me.

The consolidation

I switched to SociaVault, which puts public data from all of these behind one API and one key. My entire client became this:

const API_KEY = process.env.SOCIAVAULT_API_KEY;
const BASE = "https://api.sociavault.com";

async function sv(path, params = {}) {
  const url = new URL(BASE + path);
  Object.entries(params).forEach(([k, v]) => url.searchParams.set(k, v));
  const res = await fetch(url, { headers: { "X-API-Key": API_KEY } });
  if (!res.ok) throw new Error(`${res.status} ${await res.text()}`);
  return res.json();
}

And now pulling a profile from any platform is one line:

const tiktok = await sv("/v1/scrape/tiktok/profile", { handle: "stoolpresidente" });
const insta  = await sv("/v1/scrape/instagram/profile", { username: "natgeo" });
const x      = await sv("/v1/scrape/twitter/profile", { username: "nasa" });
const yt      = await sv("/v1/scrape/youtube/channel", { handle: "mrbeast" });

One auth header. One rate-limit model. One mental model. The responses still differ per platform (they have to — a TikTok video isn't a LinkedIn post), but everything around the data is uniform.

Being honest about the tradeoffs

This isn't free, and it's not magic. A few things worth saying:

You're paying per request. For my volume that's far cheaper than the official APIs plus my own time, but at massive scale you'd want to model it.
It's public data only. No private accounts, no DMs, no owner-only analytics like impressions. That's the correct line, and it's all I needed anyway.
If you only need one platform — say you live entirely in YouTube's API and it covers you — you might not need this at all. YouTube's official API is genuinely good.

The consolidation pays off the moment you touch two or more platforms, which most real projects do eventually.

What I built with it

Once the boundary was clean, the actual features got easy: a cross-platform profile lookup, a follower tracker, a little trend monitor. I wrote up a few of those builds if you want concrete examples — like tracking which World Cup players are blowing up and a multi-platform monitoring setup.

If you've been putting off a project because the API setup looked miserable, grab a free key (50 credits) and see how far ten lines of shared code gets you. That part genuinely surprised me.

I Was Juggling 5 Social APIs. Here's How I Got Down to One.

Olamide Olaniyan — Wed, 17 Jun 2026 09:52:00 +0000

A while back I was building a tool that needed data from Instagram, TikTok, YouTube, X, and Reddit. I started the "right" way — official APIs for each. Three weeks later I had five sets of credentials, five auth flows, five rate-limit schemes, five JSON shapes, and a permanent headache.

This is the story of how I ripped all of that out and replaced it with one API and one key. If you've ever tried to pull data from more than two platforms, you'll feel this.

The five-headed monster

Here's what "just use the official APIs" actually meant in practice:

Instagram (Meta Graph): Can only read accounts you own/manage. Want a public competitor's posts? Not happening.
TikTok: The Research API is gated behind academic approval. Commercial use? Basically no real option.
YouTube: Actually decent! Generous-ish quotas. The one bright spot.
X (Twitter): $100/month to start, and the good tiers run into thousands.
Reddit: Free tier got gutted in 2023; the paid API is pricey for what it is.

Five different mental models. Five SDKs or hand-rolled clients. Five places to break. And for half of them, the data I actually wanted (public profiles and posts) wasn't even available at any price.

What I actually needed

I wasn't doing anything exotic. I wanted public data — the stuff any logged-out user can see:

A profile's follower count and bio
A user's recent posts with engagement numbers
Search results for a keyword or hashtag
Comments on a public post

That's it. None of it requires a partnership. It's all visible in a browser. The official APIs just don't expose it (or charge a fortune for it) because they're built for managing your own presence, not for research.

The switch

I moved everything to the SociaVault API. One key, one base URL, the same request shape for every platform. My five clients collapsed into one:

const API_KEY = process.env.SOCIAVAULT_API_KEY;
const BASE = "https://api.sociavault.com";

async function sv(path, params = {}) {
  const url = new URL(BASE + path);
  Object.entries(params).forEach(([k, v]) => url.searchParams.set(k, v));
  const res = await fetch(url, { headers: { "X-API-Key": API_KEY } });
  if (!res.ok) throw new Error(`${res.status} ${await res.text()}`);
  return res.json();
}

And then every platform is just a path:

const ig      = await sv("/v1/scrape/instagram/profile", { username: "nike" });
const tiktok  = await sv("/v1/scrape/tiktok/profile", { username: "nike" });
const youtube = await sv("/v1/scrape/youtube/channel", { handle: "nike" });
const x       = await sv("/v1/scrape/twitter/profile", { username: "nike" });
const reddit  = await sv("/v1/scrape/reddit/subreddit", { subreddit: "sneakers" });

That's the whole integration. Same auth, same error handling, same retry logic for all of them. The diff in my codebase was almost cathartic — hundreds of lines of per-platform glue deleted.

The honest tradeoffs

I'm not going to pretend it's perfect for every case:

It's a paid, per-request model. You pay per call (credits). If you're doing massive volume, you'll want to do the math vs. building your own scraping infra.
It's public data only. No private accounts, no DMs, no owner-only analytics like impressions/reach. That's correct and expected, but know the boundary.
YouTube's official API is genuinely fine. If YouTube is all you need, the official one is free and good. The consolidation win shows up when you're spanning multiple platforms.

For my use case — multi-platform public data, small team, wanted to ship — it was a no-brainer. The time I got back from not maintaining five integrations paid for the credits many times over.

If you want to go deeper

I wrote a longer, no-code explainer on how these APIs actually work and where the official-vs-unofficial line sits: How Do Social Media APIs Actually Work?. There's also a piece on the hidden costs of official social media APIs that mirrors a lot of my pain here.

Free key with 50 credits at sociavault.com if you want to try collapsing your own integrations.

Anyone else been through the five-API gauntlet? Curious what finally pushed you over the edge.

Analyze the Top-Performing TikTok Ads Programmatically (TikTok Ad Library API)

Olamide Olaniyan — Tue, 16 Jun 2026 08:34:00 +0000

If you run paid social, TikTok's Creative Center is one of the most underrated tools out there. It shows you the actual top-performing ads by region, industry, and time window — ranked by real metrics like CTR, play rate, and conversions.

The catch: it's a manual browsing tool. You can't export, can't monitor over time, can't pipe it into anything. So I wired it up with the SociaVault TikTok Ad Library API and now I can pull and analyze top ads in code.

Here's a quick build.

The client

const API_KEY = process.env.SOCIAVAULT_API_KEY;
const BASE = "https://api.sociavault.com";

async function sv(path, params = {}) {
  const url = new URL(BASE + path);
  Object.entries(params).forEach(([k, v]) => url.searchParams.set(k, v));
  const res = await fetch(url, { headers: { "X-API-Key": API_KEY } });
  if (!res.ok) throw new Error(`${res.status} ${await res.text()}`);
  return res.json();
}

Step 1: Pull the top ads in your niche

Search the Creative Center's top ads, filtered and sorted however you like — here, top beauty ads in the US over the last 7 days, sorted by click-through rate:

async function topAds() {
  const { data } = await sv("/v1/scrape/tiktok-ad-library/search", {
    industry: "beauty_personal_care",
    region: "US",
    period: "7",
    order_by: "ctr",
    limit: "50",
  });
  console.log(`Found ${data.total} top ads`);
  return data.ads;
}

Each ad comes back with the creative text, CTR, likes, video download URLs, and industry/objective classification.

Step 2: Find the patterns

A quick keyword frequency pass over the top ads tells you what angles are dominating right now:

function trendingTerms(ads) {
  const counts = {};
  for (const ad of Object.values(ads)) {
    for (const word of (ad.ad_title ?? "").toLowerCase().split(/\W+/)) {
      if (word.length > 4) counts[word] = (counts[word] || 0) + 1;
    }
  }
  return Object.entries(counts).sort((a, b) => b[1] - a[1]).slice(0, 20);
}

If "peptide" or "glass" shows up in 15 of the top 50 beauty ads, that's a product angle worth testing.

Step 3: Reverse-engineer a single winner

This is the part I didn't expect. The ad detail endpoint returns a second-by-second breakdown — retention, clicks, conversions:

async function studyAd(adId) {
  const { data } = await sv("/v1/scrape/tiktok-ad-library/ad", { ad_id: adId });

  const retention = data.interactive_time_analysis?.remain;
  console.log(`Duration: ${retention.duration}s`);
  retention.analysis.forEach(p =>
    console.log(`  sec ${p.second}: ${(p.value * 100).toFixed(1)}% still watching`)
  );

  // Where do viewers actually click?
  console.log("Click hotspots:", data.interactive_time_analysis?.clicks?.highlight);
  return data;
}

The retention curve is gold. If the top ad in your category holds 55%+ at second 3 and the click spike lands at second 16, you know exactly what hook strength and CTA timing to aim for.

Cost

One credit per search (returns up to 50 ads), one credit per ad detail lookup. A full weekly sweep across a few industries is pennies.

Honest take on alternatives

Tools like PiPiADS, Minea, and BigSpy do similar things with a nicer UI, but they charge $100–300/month and most can't show you the retention curves. If you want a polished dashboard and don't code, those are fine. If you want raw, exportable data and the per-second analysis, the API route is cheaper and richer.

Going further

I wrote up the full creative-research workflow here: TikTok Ad Library API: Research Competitor Ads at Scale and a deeper dive on using retention curves to optimize hooks.

Free key gets you 50 credits — enough to sweep your whole industry and study a dozen winners. Curious what angles are dominating your niche right now?

What YouTube Comment Replies Reveal Before User Interviews Do

Olamide Olaniyan — Sat, 09 May 2026 07:20:00 +0000

Most product teams stop at top-level comments.

That is usually where the obvious reactions live.

Replies are where people explain themselves.

That is a much better place to look if you care about:

objections
feature requests
product confusion
pricing resistance
onboarding friction
competitor comparisons

I still like user interviews. I still run them. I would never tell you YouTube comments should replace talking to real customers.

But if you are trying to find language, recurring problems, and messy real-world buyer questions before you book ten calls, YouTube comment replies are a very underused source.

This post shows you how I use them for product research, how to pull them with JavaScript and Python, and when this approach is smarter than interviews, surveys, or the official API.

Why Replies Matter More Than Top-Level Comments

Top-level comments are often performative.

People praise the creator. Joke about the video. Drop a quick take. Argue with the premise.

Replies are different.

Replies are where someone says:

"I tried this and got stuck at the migration step"
"This sounds good until you have to bring in a team"
"We switched away from this because the pricing got weird"
"Does this work if you need approval flows too?"

That is product language.

Not polished landing-page language. Not survey language. Not customer-success-call language.

Real language.

That is why I keep coming back to replies.

The Research Question Comes First

Do not start by scraping thousands of replies and hoping insight appears.

Start with a question.

Examples:

What objections keep showing up around this category?
What edge cases do people care about most?
What makes buyers hesitate?
What pain points do competitor users describe in their own words?

If you have that question first, the reply threads become much easier to read.

JavaScript Version: Pull Comments, Then Pull Replies

With YouTube, the workflow is straightforward.

First fetch top-level comments for a video. Then look for comments that include a repliesContinuationToken. That token is the key to the reply thread.

const headers = {
  'X-API-Key': process.env.SOCIAVAULT_API_KEY,
};

async function fetchJson(url) {
  const response = await fetch(url, { headers });
  if (!response.ok) {
    throw new Error(`Request failed with ${response.status}`);
  }
  return response.json();
}

async function getCommentThreads(videoUrl) {
  const commentsJson = await fetchJson(
    `https://api.sociavault.com/v1/scrape/youtube/video/comments?url=${encodeURIComponent(videoUrl)}&order=top`
  );

  const comments = Object.values(commentsJson.data?.comments || {});
  const threads = [];

  for (const comment of comments) {
    const token = comment.repliesContinuationToken;
    if (!token) continue;

    const repliesJson = await fetchJson(
      `https://api.sociavault.com/v1/scrape/youtube/video/comment-replies?continuationToken=${encodeURIComponent(token)}`
    );

    threads.push({
      topLevelComment: comment.content || comment.text,
      replies: repliesJson.data?.comments || repliesJson.data || [],
    });
  }

  return threads;
}

function classifyReply(text) {
  const value = (text || '').toLowerCase();

  if (/(too expensive|pricing|cost|price)/.test(value)) return 'pricing';
  if (/(integrat|api|sync|connect)/.test(value)) return 'integration';
  if (/(confus|hard|complicated|setup)/.test(value)) return 'onboarding';
  if (/(competitor|alternative|instead of|switched from)/.test(value)) return 'competitor';
  return 'other';
}

async function researchVideo(videoUrl) {
  const threads = await getCommentThreads(videoUrl);
  const summary = {};

  for (const thread of threads) {
    for (const reply of thread.replies) {
      const text = reply.content || reply.text || '';
      const label = classifyReply(text);
      summary[label] = summary[label] || [];
      summary[label].push(text);
    }
  }

  return summary;
}

researchVideo('https://www.youtube.com/watch?v=dQw4w9WgXcQ')
  .then(summary => console.log(summary))
  .catch(error => console.error(error));

That already gives you something useful: a rough thematic breakdown of reply language for one video.

Do that across five or ten relevant videos and patterns start showing up fast.

If you want a simpler way to get the public YouTube data without managing the collection layer, SociaVault is what I would use here.

Python Version: Same Workflow, Easy to Batch

Python is great for this kind of batch research workflow.

import os
import requests


HEADERS = {'X-API-Key': os.environ['SOCIAVAULT_API_KEY']}


def fetch_json(url):
    response = requests.get(url, headers=HEADERS, timeout=30)
    response.raise_for_status()
    return response.json()


def get_comment_threads(video_url):
    comments_json = fetch_json(
        f'https://api.sociavault.com/v1/scrape/youtube/video/comments?url={video_url}&order=top'
    )

    comments = list((comments_json.get('data') or {}).get('comments', {}).values())
    threads = []

    for comment in comments:
        token = comment.get('repliesContinuationToken')
        if not token:
            continue

        replies_json = fetch_json(
            f'https://api.sociavault.com/v1/scrape/youtube/video/comment-replies?continuationToken={token}'
        )

        threads.append({
            'topLevelComment': comment.get('content') or comment.get('text'),
            'replies': replies_json.get('data', {}).get('comments') or replies_json.get('data') or [],
        })

    return threads


def classify_reply(text):
    value = (text or '').lower()

    if any(term in value for term in ['too expensive', 'pricing', 'cost', 'price']):
        return 'pricing'
    if any(term in value for term in ['integrat', 'api', 'sync', 'connect']):
        return 'integration'
    if any(term in value for term in ['confus', 'hard', 'complicated', 'setup']):
        return 'onboarding'
    if any(term in value for term in ['competitor', 'alternative', 'instead of', 'switched from']):
        return 'competitor'
    return 'other'


def research_video(video_url):
    threads = get_comment_threads(video_url)
    summary = {}

    for thread in threads:
        for reply in thread['replies']:
            text = reply.get('content') or reply.get('text') or ''
            label = classify_reply(text)
            summary.setdefault(label, []).append(text)

    return summary


summary = research_video('https://www.youtube.com/watch?v=dQw4w9WgXcQ')
for label, examples in summary.items():
    print(f'\n## {label.upper()}')
    for example in examples[:5]:
        print('-', example)

The nice thing about the Python version is that it is very easy to turn into a weekly batch job that analyzes several videos in one pass.

What To Do With the Reply Data

This is the part people often skip.

Do not just collect replies.

Turn them into something reusable.

My usual workflow is:

pick 5-10 relevant videos in the category
pull top-level comments
fetch reply threads where discussion actually happened
tag replies by theme
save exact phrases, not just summaries

That fifth step matters a lot.

You want the real wording.

If ten people say some version of:

looks useful, but this feels too complicated for a small team

that phrase is often more valuable than a neat internal note like "SMB onboarding concerns exist."

The messy language is usually the useful language.

Where This Is Better Than Interviews

This is better than user interviews when:

you need fast raw language
you are early and need hypothesis fuel
you want to study a competitor category before reaching out to users
you want public objections at scale

It is especially useful before writing:

landing page copy
pricing FAQs
comparison pages
ad hooks
onboarding messaging

Where Interviews Still Win

This is where I want to be very clear.

YouTube replies are not a replacement for talking to users.

Interviews are still much better when you need:

follow-up questions
segment-specific nuance
willingness-to-pay signals
workflow details that people do not write publicly
actual product validation

Think of reply analysis as the thing that helps you ask better interview questions.

Not the thing that replaces interviews.

Honest Alternatives

There are three realistic options here.

The official YouTube API

Good if you are already comfortable inside Google's ecosystem and only need YouTube.

Manual reading

Good for quick one-off research.

Bad if you want something repeatable or cross-video.

Public data workflow with SociaVault

Best when you want a simpler API flow and expect this research to become part of a larger stack.

That is usually where I land, because I rarely want a one-platform-only solution for very long.

Final Take

Top-level comments tell you the reaction.

Replies tell you the reasoning.

That is why I keep using them for product research, competitor analysis, copywriting, and objection mining.

If you want a practical way to turn YouTube reply threads into reusable research, SociaVault gives you the public data layer so you can spend your time on analysis instead of collection.

Start small. Pick five videos. Tag the replies. Save the exact phrases.

You will usually get better language out of that exercise than out of a week of guessing.

webdev #youtube #productivity #python #javascript

How I'd Vet Creators for Whitelisting and Paid Media Access

Olamide Olaniyan — Fri, 08 May 2026 21:19:00 +0000

Whitelisting is where creator vetting gets more serious.

A normal sponsorship asks, "Should we work with this person?"

Whitelisting asks a more expensive question:

should we put paid spend behind this creator's identity, content style, and public trust?

That is a higher bar.

And a lot of teams still evaluate it with roughly the same checklist they use for ordinary creator partnerships.

I would not.

Because once paid media is involved, weak judgment gets amplified.

The wrong creator is not just a soft underperformer anymore. They become a scaled distribution problem.

This is the review workflow I would build before approving any creator for whitelisting or broader paid media access.

The Core Difference

For a standard partnership, decent content fit might be enough.

For whitelisting, I want stronger answers to four things:

does the creator feel credible on camera?
does their recent content look stable and brand-safe?
does their audience engagement look believable enough?
can I imagine running spend through this identity without regret?

That last question matters more than people admit.

The Risk Areas I Care About

The first-pass vetting layer I want includes:

1. Creative consistency

If the recent content is chaotic, stale, or all over the place, that matters.

2. Brand safety

Not just obvious harmful content. Also erratic tone, sloppy claims, or recurring controversy patterns.

3. Audience reliability

Paid media does not fix weak trust.

4. Operational consistency

If posting activity is inconsistent or the content mix shifts wildly, that is a risk too.

JavaScript Version: First-Pass Whitelisting Readiness Check

This uses recent TikTok videos as the public signal layer, then scores readiness conservatively.

const headers = {
  'X-API-Key': process.env.SOCIAVAULT_API_KEY,
};

async function fetchJson(url) {
  const response = await fetch(url, { headers });
  if (!response.ok) {
    throw new Error(`Request failed with ${response.status}`);
  }
  return response.json();
}

function average(values) {
  if (!values.length) return 0;
  return values.reduce((sum, value) => sum + value, 0) / values.length;
}

function containsRiskyLanguage(text = '') {
  return /(hate|scam|guaranteed income|get rich quick|offensive|explicit)/i.test(text);
}

function meaningfulCaption(text = '') {
  return (text || '').trim().length >= 12;
}

function buildDecision(score) {
  if (score >= 75) return 'approve';
  if (score >= 55) return 'caution';
  return 'decline';
}

async function assessWhitelisting(handle) {
  const [profileJson, videosJson, followersJson] = await Promise.all([
    fetchJson(`https://api.sociavault.com/v1/scrape/tiktok/profile?handle=${encodeURIComponent(handle)}`),
    fetchJson(`https://api.sociavault.com/v1/scrape/tiktok/videos?handle=${encodeURIComponent(handle)}&amount=12&sort_by=latest&trim=true`),
    fetchJson(`https://api.sociavault.com/v1/scrape/tiktok/followers?handle=${encodeURIComponent(handle)}&trim=true`),
  ]);

  const profile = profileJson.data || {};
  const videos = videosJson.data || [];
  const followers = followersJson.data || [];

  const followerCount = profile.stats?.followerCount || profile.followerCount || 0;
  const engagements = videos.map(video => (video.stats?.diggCount || 0) + (video.stats?.commentCount || 0));
  const engagementRate = followerCount > 0 ? (average(engagements) / followerCount) * 100 : 0;

  const brandSafetyFlags = videos.filter(video => containsRiskyLanguage(video.desc || '')).length;
  const weakCaptionShare = videos.length
    ? videos.filter(video => !meaningfulCaption(video.desc || '')).length / videos.length
    : 0;

  const suspiciousFollowerShare = followers.length
    ? followers.filter(follower => {
        const username = follower.username || follower.unique_id || '';
        return /\d{5,}/.test(username);
      }).length / followers.length
    : 0;

  let score = 100;
  if (engagementRate < 1) score -= 20;
  if (brandSafetyFlags > 0) score -= 25;
  if (weakCaptionShare > 0.5) score -= 15;
  if (suspiciousFollowerShare > 0.3) score -= 20;
  if (videos.length < 8) score -= 10;

  return {
    handle,
    followerCount,
    engagementRate: Number(engagementRate.toFixed(2)),
    brandSafetyFlags,
    weakCaptionShare: Number(weakCaptionShare.toFixed(2)),
    suspiciousFollowerShare: Number(suspiciousFollowerShare.toFixed(2)),
    recentVideoCount: videos.length,
    score,
    decision: buildDecision(score),
  };
}

assessWhitelisting('creator_handle')
  .then(report => console.log(report))
  .catch(error => console.error(error));

This is not a legal approval system.

It is a strong first-pass screen that stops weak candidates from getting too far.

Python Version: Batch Whitelisting Reviews for Creator Lists

If I were screening a campaign list, I would probably use Python.

import os
import re
import requests


HEADERS = {'X-API-Key': os.environ['SOCIAVAULT_API_KEY']}


def fetch_json(url):
    response = requests.get(url, headers=HEADERS, timeout=30)
    response.raise_for_status()
    return response.json()


def average(values):
    return sum(values) / len(values) if values else 0


def contains_risky_language(text=''):
    return bool(re.search(r'(hate|scam|guaranteed income|get rich quick|offensive|explicit)', text, re.IGNORECASE))


def meaningful_caption(text=''):
    return len((text or '').strip()) >= 12


def build_decision(score):
    if score >= 75:
        return 'approve'
    if score >= 55:
        return 'caution'
    return 'decline'


def assess_whitelisting(handle):
    profile_json = fetch_json(f'https://api.sociavault.com/v1/scrape/tiktok/profile?handle={handle}')
    videos_json = fetch_json(f'https://api.sociavault.com/v1/scrape/tiktok/videos?handle={handle}&amount=12&sort_by=latest&trim=true')
    followers_json = fetch_json(f'https://api.sociavault.com/v1/scrape/tiktok/followers?handle={handle}&trim=true')

    profile = profile_json.get('data') or {}
    videos = videos_json.get('data') or []
    followers = followers_json.get('data') or []

    follower_count = profile.get('stats', {}).get('followerCount') or profile.get('followerCount') or 0
    engagements = [(video.get('stats', {}).get('diggCount', 0) + video.get('stats', {}).get('commentCount', 0)) for video in videos]
    engagement_rate = (average(engagements) / follower_count * 100) if follower_count else 0

    brand_safety_flags = sum(1 for video in videos if contains_risky_language(video.get('desc') or ''))
    weak_caption_share = (sum(1 for video in videos if not meaningful_caption(video.get('desc') or '')) / len(videos)) if videos else 0
    suspicious_follower_share = (
        sum(1 for follower in followers if re.search(r'\d{5,}', follower.get('username') or follower.get('unique_id') or '')) / len(followers)
        if followers else 0
    )

    score = 100
    if engagement_rate < 1:
        score -= 20
    if brand_safety_flags > 0:
        score -= 25
    if weak_caption_share > 0.5:
        score -= 15
    if suspicious_follower_share > 0.3:
        score -= 20
    if len(videos) < 8:
        score -= 10

    return {
        'handle': handle,
        'followerCount': follower_count,
        'engagementRate': round(engagement_rate, 2),
        'brandSafetyFlags': brand_safety_flags,
        'weakCaptionShare': round(weak_caption_share, 2),
        'suspiciousFollowerShare': round(suspicious_follower_share, 2),
        'recentVideoCount': len(videos),
        'score': score,
        'decision': build_decision(score),
    }


handles = ['creator_one', 'creator_two', 'creator_three']
reports = [assess_whitelisting(handle) for handle in handles]
print(reports)

What This Workflow Does Not Replace

This part matters.

Whitelisting approval should still include manual review for:

contract scope
usage rights
platform access details
final brand-safety judgment
campaign-specific creative fit

The point of the workflow is not to automate those decisions away.

The point is to stop weak candidates earlier.

Honest Alternatives

You can run this process entirely manually. Plenty of teams still do.

But if you are screening real creator volume, manual whitelisting review becomes inconsistent very quickly.

You can also buy a full creator platform, which may be the right move for larger teams.

I like a lighter setup when the actual value is in the review logic itself. In that model, SociaVault gives me the public social data layer and I keep the approval rules in my own workflow.

That is a cleaner separation.

Final Take

Whitelisting deserves a stricter review standard than a normal creator collaboration.

That is the main idea.

If you are about to put spend behind a creator's identity, you want more than a nice profile and a decent average view count.

You want evidence that the content is stable, the engagement is believable, the recent output is usable, and the risk level is acceptable.

If you want to build that pre-approval layer without wiring the public data collection stack from scratch, SociaVault is a good place to start.

Then keep the review honest and conservative.

That is usually the difference between a scalable paid-media workflow and a very expensive lesson.

webdev #python #javascript #marketing #creatoreconomy

I Tried to Normalize 8 Social Platforms Into One JSON Schema. Here's What Broke

Olamide Olaniyan — Thu, 07 May 2026 19:19:00 +0000

I thought this was going to be easy.

Take data from TikTok, Instagram, YouTube, Reddit, LinkedIn, Threads, X, and Facebook. Map it into one clean shape. Ship it. Move on.

That lasted about a day.

By day two, I had already learned the first ugly truth of social data engineering:

the hard part is not collecting the data. The hard part is deciding what the same thing even means across platforms.

"Likes" are not always likes. "Views" are not always views. Some platforms expose shares publicly, some do not. One platform gives you createTime, another gives you ISO timestamps, another gives you nested objects with three different possible IDs depending on the endpoint.

If you're building a social dashboard, creator analytics tool, moderation workflow, or competitor monitoring system, you hit this wall fast.

So this post is the version I wish I had read earlier: what broke, what schema I kept, what I stopped trying to normalize, and how I now build a social media JSON schema without lying to myself.

The Mistake I Made First

My first schema looked clean.

{
  "id": "123",
  "platform": "tiktok",
  "author": "creator_handle",
  "text": "post text",
  "likes": 100,
  "comments": 12,
  "views": 5000,
  "shares": 4,
  "published_at": "2026-04-24T12:00:00Z"
}

That shape is attractive because it feels universal.

It is also incomplete in exactly the ways that cause expensive bugs later.

Here is what broke almost immediately:

TikTok and Instagram expose author and stats differently.
YouTube comments do not behave like short-form video posts.
Reddit has score, comments, and post metadata that do not map cleanly to "likes/views/shares."
LinkedIn often has less public engagement detail than people expect.
Some endpoints return counts as nested stats objects. Others flatten them.
URLs and canonical IDs vary wildly by platform.
Missing values are common, and missing does not mean zero.

That last point matters a lot.

If a platform does not expose a public field, setting it to 0 is false precision. You are making downstream analysis worse while pretending you are making it simpler.

The Rule That Saved Me

I stopped trying to force every platform into the exact same semantic model.

Instead, I switched to this rule:

Normalize the envelope. Preserve the raw meaning.

That means every object should share a predictable wrapper, but platform-specific nuance should stay visible.

The schema I ended up liking looks more like this:

{
  "platform": "tiktok",
  "entity_type": "post",
  "external_id": "735902002991",
  "canonical_url": "https://www.tiktok.com/@creator/video/735902002991",
  "author": {
    "handle": "creator",
    "display_name": "Creator Name"
  },
  "content": {
    "text": "caption text",
    "media_type": "video"
  },
  "metrics": {
    "likes": 1200,
    "comments": 87,
    "shares": 41,
    "views": 54000
  },
  "published_at": "2026-04-24T12:00:00Z",
  "availability": {
    "views": "public",
    "shares": "public",
    "saves": "not_available"
  },
  "raw": {}
}

Notice what changed:

metrics is grouped instead of flattened.
availability tells downstream code whether a field is real, missing, or unavailable.
raw is preserved for debugging, reprocessing, and future schema changes.

This is much less elegant than the first version.

It is also much more survivable.

What I Normalize Now

These are the fields I consider worth normalizing across platforms:

platform
entity_type
external_id
canonical_url
author.handle
author.display_name
content.text
content.media_type
metrics.likes
metrics.comments
metrics.views
metrics.shares
published_at

That is enough to make cross-platform dashboards, alerts, ranking jobs, and exports practical.

But I do not pretend every field exists everywhere.

JavaScript Version: A Normalizer That Stays Honest

This example is intentionally boring. That is a good sign.

If your normalization layer is too clever, it becomes unmaintainable fast.

function pickFirst(...values) {
  return values.find(value => value !== undefined && value !== null && value !== '');
}

function toNumber(value) {
  const number = Number(value);
  return Number.isFinite(number) ? number : null;
}

function toIsoDate(value) {
  if (!value) return null;

  if (typeof value === 'number') {
    return new Date(value * 1000).toISOString();
  }

  const parsed = new Date(value);
  return Number.isNaN(parsed.getTime()) ? null : parsed.toISOString();
}

function normalizePost(platform, raw) {
  const base = {
    platform,
    entity_type: 'post',
    external_id: null,
    canonical_url: null,
    author: {
      handle: null,
      display_name: null,
    },
    content: {
      text: null,
      media_type: null,
    },
    metrics: {
      likes: null,
      comments: null,
      views: null,
      shares: null,
    },
    published_at: null,
    availability: {
      likes: 'unknown',
      comments: 'unknown',
      views: 'unknown',
      shares: 'unknown',
    },
    raw,
  };

  if (platform === 'tiktok') {
    return {
      ...base,
      external_id: pickFirst(raw.id, raw.aweme_id),
      canonical_url: pickFirst(raw.share_url, raw.url),
      author: {
        handle: pickFirst(raw.author?.uniqueId, raw.author?.handle),
        display_name: pickFirst(raw.author?.nickname, raw.author?.displayName),
      },
      content: {
        text: pickFirst(raw.desc, raw.caption),
        media_type: 'video',
      },
      metrics: {
        likes: toNumber(pickFirst(raw.stats?.diggCount, raw.like_count)),
        comments: toNumber(pickFirst(raw.stats?.commentCount, raw.comment_count)),
        views: toNumber(pickFirst(raw.stats?.playCount, raw.view_count)),
        shares: toNumber(pickFirst(raw.stats?.shareCount, raw.share_count)),
      },
      published_at: toIsoDate(pickFirst(raw.createTime, raw.created_at)),
      availability: {
        likes: 'public',
        comments: 'public',
        views: 'public',
        shares: 'public',
      },
    };
  }

  if (platform === 'youtube_comment') {
    return {
      ...base,
      entity_type: 'comment',
      external_id: pickFirst(raw.commentId, raw.id),
      canonical_url: pickFirst(raw.url),
      author: {
        handle: pickFirst(raw.author?.channelHandle, raw.author?.name),
        display_name: pickFirst(raw.author?.name, raw.authorDisplayName),
      },
      content: {
        text: pickFirst(raw.content, raw.text),
        media_type: 'text',
      },
      metrics: {
        likes: toNumber(pickFirst(raw.likes, raw.likeCount)),
        comments: toNumber(pickFirst(raw.replyCount, raw.repliesCount)),
        views: null,
        shares: null,
      },
      published_at: toIsoDate(pickFirst(raw.publishedAt, raw.timestamp)),
      availability: {
        likes: 'public',
        comments: 'public',
        views: 'not_available',
        shares: 'not_available',
      },
    };
  }

  if (platform === 'reddit') {
    return {
      ...base,
      external_id: pickFirst(raw.id, raw.post_id),
      canonical_url: pickFirst(raw.permalink, raw.url),
      author: {
        handle: pickFirst(raw.author, raw.author_name),
        display_name: pickFirst(raw.author, raw.author_name),
      },
      content: {
        text: pickFirst(raw.selftext, raw.title),
        media_type: raw.is_video ? 'video' : 'text',
      },
      metrics: {
        likes: toNumber(pickFirst(raw.score, raw.upvotes)),
        comments: toNumber(pickFirst(raw.num_comments, raw.comment_count)),
        views: null,
        shares: null,
      },
      published_at: toIsoDate(pickFirst(raw.created_utc, raw.created_at)),
      availability: {
        likes: 'public_proxy',
        comments: 'public',
        views: 'not_available',
        shares: 'not_available',
      },
    };
  }

  throw new Error(`Unsupported platform: ${platform}`);
}

const tiktokExample = {
  id: '735902002991',
  desc: 'How I batch content research in 15 minutes',
  createTime: 1713936000,
  author: { uniqueId: 'creator_handle', nickname: 'Creator Name' },
  stats: { diggCount: 1200, commentCount: 87, playCount: 54000, shareCount: 41 },
  share_url: 'https://www.tiktok.com/@creator_handle/video/735902002991',
};

console.log(normalizePost('tiktok', tiktokExample));

The important part is not the exact shape above.

The important part is that unsupported or unavailable fields stay explicit.

That makes your downstream analytics much easier to trust.

Python Version: Same Strategy, Different Runtime

If your data jobs live in Python, the same idea applies.

from datetime import datetime, timezone


def pick_first(*values):
    for value in values:
        if value not in (None, '', []):
            return value
    return None


def to_number(value):
    try:
        return int(value)
    except (TypeError, ValueError):
        return None


def to_iso_date(value):
    if value is None:
        return None

    if isinstance(value, (int, float)):
        return datetime.fromtimestamp(value, tz=timezone.utc).isoformat()

    try:
        return datetime.fromisoformat(str(value).replace('Z', '+00:00')).isoformat()
    except ValueError:
        return None


def normalize_post(platform, raw):
    base = {
        'platform': platform,
        'entity_type': 'post',
        'external_id': None,
        'canonical_url': None,
        'author': {
            'handle': None,
            'display_name': None,
        },
        'content': {
            'text': None,
            'media_type': None,
        },
        'metrics': {
            'likes': None,
            'comments': None,
            'views': None,
            'shares': None,
        },
        'published_at': None,
        'availability': {
            'likes': 'unknown',
            'comments': 'unknown',
            'views': 'unknown',
            'shares': 'unknown',
        },
        'raw': raw,
    }

    if platform == 'tiktok':
        return {
            **base,
            'external_id': pick_first(raw.get('id'), raw.get('aweme_id')),
            'canonical_url': pick_first(raw.get('share_url'), raw.get('url')),
            'author': {
                'handle': pick_first(raw.get('author', {}).get('uniqueId'), raw.get('author', {}).get('handle')),
                'display_name': pick_first(raw.get('author', {}).get('nickname'), raw.get('author', {}).get('displayName')),
            },
            'content': {
                'text': pick_first(raw.get('desc'), raw.get('caption')),
                'media_type': 'video',
            },
            'metrics': {
                'likes': to_number(pick_first(raw.get('stats', {}).get('diggCount'), raw.get('like_count'))),
                'comments': to_number(pick_first(raw.get('stats', {}).get('commentCount'), raw.get('comment_count'))),
                'views': to_number(pick_first(raw.get('stats', {}).get('playCount'), raw.get('view_count'))),
                'shares': to_number(pick_first(raw.get('stats', {}).get('shareCount'), raw.get('share_count'))),
            },
            'published_at': to_iso_date(pick_first(raw.get('createTime'), raw.get('created_at'))),
            'availability': {
                'likes': 'public',
                'comments': 'public',
                'views': 'public',
                'shares': 'public',
            },
        }

    if platform == 'youtube_comment':
        return {
            **base,
            'entity_type': 'comment',
            'external_id': pick_first(raw.get('commentId'), raw.get('id')),
            'canonical_url': raw.get('url'),
            'author': {
                'handle': pick_first(raw.get('author', {}).get('channelHandle'), raw.get('author', {}).get('name')),
                'display_name': pick_first(raw.get('author', {}).get('name'), raw.get('authorDisplayName')),
            },
            'content': {
                'text': pick_first(raw.get('content'), raw.get('text')),
                'media_type': 'text',
            },
            'metrics': {
                'likes': to_number(pick_first(raw.get('likes'), raw.get('likeCount'))),
                'comments': to_number(pick_first(raw.get('replyCount'), raw.get('repliesCount'))),
                'views': None,
                'shares': None,
            },
            'published_at': to_iso_date(pick_first(raw.get('publishedAt'), raw.get('timestamp'))),
            'availability': {
                'likes': 'public',
                'comments': 'public',
                'views': 'not_available',
                'shares': 'not_available',
            },
        }

    if platform == 'reddit':
        return {
            **base,
            'external_id': pick_first(raw.get('id'), raw.get('post_id')),
            'canonical_url': pick_first(raw.get('permalink'), raw.get('url')),
            'author': {
                'handle': pick_first(raw.get('author'), raw.get('author_name')),
                'display_name': pick_first(raw.get('author'), raw.get('author_name')),
            },
            'content': {
                'text': pick_first(raw.get('selftext'), raw.get('title')),
                'media_type': 'video' if raw.get('is_video') else 'text',
            },
            'metrics': {
                'likes': to_number(pick_first(raw.get('score'), raw.get('upvotes'))),
                'comments': to_number(pick_first(raw.get('num_comments'), raw.get('comment_count'))),
                'views': None,
                'shares': None,
            },
            'published_at': to_iso_date(pick_first(raw.get('created_utc'), raw.get('created_at'))),
            'availability': {
                'likes': 'public_proxy',
                'comments': 'public',
                'views': 'not_available',
                'shares': 'not_available',
            },
        }

    raise ValueError(f'Unsupported platform: {platform}')


tiktok_example = {
    'id': '735902002991',
    'desc': 'How I batch content research in 15 minutes',
    'createTime': 1713936000,
    'author': {'uniqueId': 'creator_handle', 'nickname': 'Creator Name'},
    'stats': {'diggCount': 1200, 'commentCount': 87, 'playCount': 54000, 'shareCount': 41},
    'share_url': 'https://www.tiktok.com/@creator_handle/video/735902002991',
}

print(normalize_post('tiktok', tiktok_example))

What I Stopped Trying to Normalize

This part saved me more time than any helper function.

There are fields I no longer force into a universal schema unless the use case truly needs them:

saves/bookmarks
watch time
audience demographics
thread depth nuances
platform-specific moderation states
estimated reach proxies

Why?

Because those fields mean different things, appear inconsistently, or are missing often enough that the abstraction becomes misleading.

If a product feature really depends on one of those platform-specific concepts, I expose that field under a platform namespace instead.

That keeps the shared schema clean and the specialized data honest.

Honest Alternatives

There are three valid approaches here.

1. Normalize everything into one schema

Good for dashboards and cross-platform ranking.

Bad if you start pretending unavailable fields are universal.

2. Keep completely separate per-platform models

Good for correctness.

Bad for product teams that want a single report, feed, or analytics layer.

3. Keep a thin shared schema plus raw JSON

This is the one I use most often.

It gives you enough consistency for application logic without throwing away nuance.

If you are early, start there.

Where SociaVault Fits

This is exactly where a data layer like SociaVault helps.

I do not want to spend my week maintaining eight different scraping stacks just to get to the actual product problem, which is normalization, ranking, alerting, and reporting.

So my usual split is:

use SociaVault for the public social data collection layer
normalize into my own schema
keep the raw payload beside the normalized object

That keeps the engineering work focused on product logic instead of collection plumbing.

Final Take

Trying to build a universal social media JSON schema taught me something useful: the goal is not to erase platform differences.

The goal is to make those differences manageable.

Normalize the envelope. Preserve the raw payload. Mark unavailable fields honestly. Avoid fake precision.

If you are building social dashboards, creator analytics, competitor tracking, or moderation tools, that approach will hold up much better than the "one perfect schema" fantasy.

And if you want to skip the collection layer and spend your time on the normalization layer instead, start with SociaVault.

webdev #api #json #datascience #javascript

What I Learned Building a Diff Engine for Social Profiles, Posts, and Follower Counts

Olamide Olaniyan — Wed, 06 May 2026 22:17:00 +0000

The first version of my social monitoring system was technically correct and practically useless.

Every poll produced changes.

Follower count up by 3.

Bio spacing changed.

A post metric moved slightly.

A timestamp field came back in a different format.

The system was full of "change," and almost none of it mattered.

That was when I realized a social monitoring product does not really need polling logic first.

It needs a good diff engine.

Because if you cannot decide what changed meaningfully, the rest of the stack becomes noise generation.

So this is the diff model I use now for profiles, posts, and follower counts: what I compare, what I ignore, how I implement it in JavaScript and Python, and where a public social data layer like SociaVault makes the whole workflow much easier.

What a Good Diff Engine Actually Does

I think of a diff engine as having three jobs:

normalize the old and new state into comparable shapes
detect meaningful differences
classify those differences by importance

That third part is the one most implementations skip.

A change is not automatically an event.

That distinction is the difference between useful monitoring and alert fatigue.

The Three Objects I Compare Most Often

In social systems, I usually diff three categories.

1. Profiles

follower count
following count
bio
display name
profile image URL
post count or video count

2. Posts

caption or body text
likes
comments
views
pinned status if available
availability or deletion state

3. Collections

newest post IDs
newest comment IDs
active ad signatures
landing page URL sets

The rules are different for each one.

That matters.

My Threshold Rule

This one changed the quality of my alerts immediately:

numeric fields need thresholds. text fields need normalization. collection fields need set comparison.

If you diff all three types the same way, you get garbage.

For example:

follower changes need percentage or absolute thresholds
bios should be trimmed and whitespace-normalized first
post IDs should be compared as sets
metric deltas should often be classified, not just detected

That is the whole trick.

JavaScript Version: Diffing Social State Without Lying

This is the kind of diff function I like in Node services.

function normalizeText(value) {
  return (value || '').replace(/\s+/g, ' ').trim();
}

function diffNumber(previous, current, options = {}) {
  const { minAbsolute = 0, minPercent = 0 } = options;
  const prev = Number(previous || 0);
  const curr = Number(current || 0);
  const delta = curr - prev;
  const percent = prev > 0 ? Math.abs(delta / prev) * 100 : 0;

  const changed = Math.abs(delta) >= minAbsolute || percent >= minPercent;

  return {
    changed,
    previous: prev,
    current: curr,
    delta,
    percent: Number(percent.toFixed(2)),
  };
}

function diffText(previous, current) {
  const prev = normalizeText(previous);
  const curr = normalizeText(current);

  return {
    changed: prev !== curr,
    previous: prev,
    current: curr,
  };
}

function diffSet(previous = [], current = []) {
  const prev = new Set(previous);
  const curr = new Set(current);

  const added = [...curr].filter(item => !prev.has(item));
  const removed = [...prev].filter(item => !curr.has(item));

  return {
    changed: added.length > 0 || removed.length > 0,
    added,
    removed,
  };
}

function diffProfile(previous, current) {
  const followers = diffNumber(previous.followers, current.followers, {
    minAbsolute: 100,
    minPercent: 5,
  });

  const posts = diffNumber(previous.posts, current.posts, {
    minAbsolute: 1,
  });

  const bio = diffText(previous.bio, current.bio);
  const displayName = diffText(previous.displayName, current.displayName);

  const changes = [];

  if (followers.changed) {
    changes.push({ type: 'follower_change', ...followers, severity: 'medium' });
  }

  if (posts.changed && posts.delta > 0) {
    changes.push({ type: 'new_post_count', ...posts, severity: 'high' });
  }

  if (bio.changed) {
    changes.push({ type: 'bio_updated', ...bio, severity: 'low' });
  }

  if (displayName.changed) {
    changes.push({ type: 'display_name_updated', ...displayName, severity: 'low' });
  }

  return changes;
}

const previousProfile = {
  followers: 10240,
  posts: 84,
  bio: 'Helping creators grow faster',
  displayName: 'Creator Ops',
};

const currentProfile = {
  followers: 10980,
  posts: 85,
  bio: 'Helping creators grow faster with better workflows',
  displayName: 'Creator Ops',
};

console.log(diffProfile(previousProfile, currentProfile));

That pattern scales well because each diff type stays simple.

Then you can route the resulting changes into alerts, digests, dashboards, or logs.

Python Version: Same Diff Model, Easy to Batch

In Python I use almost the same logic, just in a more batch-friendly format.

def normalize_text(value):
    return ' '.join((value or '').split())


def diff_number(previous, current, min_absolute=0, min_percent=0):
    prev = float(previous or 0)
    curr = float(current or 0)
    delta = curr - prev
    percent = abs(delta / prev) * 100 if prev > 0 else 0

    changed = abs(delta) >= min_absolute or percent >= min_percent

    return {
        'changed': changed,
        'previous': prev,
        'current': curr,
        'delta': delta,
        'percent': round(percent, 2),
    }


def diff_text(previous, current):
    prev = normalize_text(previous)
    curr = normalize_text(current)
    return {
        'changed': prev != curr,
        'previous': prev,
        'current': curr,
    }


def diff_set(previous=None, current=None):
    prev = set(previous or [])
    curr = set(current or [])

    added = sorted(curr - prev)
    removed = sorted(prev - curr)

    return {
        'changed': bool(added or removed),
        'added': added,
        'removed': removed,
    }


def diff_profile(previous, current):
    followers = diff_number(previous.get('followers'), current.get('followers'), min_absolute=100, min_percent=5)
    posts = diff_number(previous.get('posts'), current.get('posts'), min_absolute=1)
    bio = diff_text(previous.get('bio'), current.get('bio'))
    display_name = diff_text(previous.get('displayName'), current.get('displayName'))

    changes = []

    if followers['changed']:
        changes.append({'type': 'follower_change', 'severity': 'medium', **followers})

    if posts['changed'] and posts['delta'] > 0:
        changes.append({'type': 'new_post_count', 'severity': 'high', **posts})

    if bio['changed']:
        changes.append({'type': 'bio_updated', 'severity': 'low', **bio})

    if display_name['changed']:
        changes.append({'type': 'display_name_updated', 'severity': 'low', **display_name})

    return changes


previous_profile = {
    'followers': 10240,
    'posts': 84,
    'bio': 'Helping creators grow faster',
    'displayName': 'Creator Ops',
}

current_profile = {
    'followers': 10980,
    'posts': 85,
    'bio': 'Helping creators grow faster with better workflows',
    'displayName': 'Creator Ops',
}

print(diff_profile(previous_profile, current_profile))

The Most Important Thing: Severity Classification

This is where the diff engine becomes a monitoring system instead of just a comparison utility.

I try to classify changes like this:

low: bio text, display name formatting, minor stat movement
medium: meaningful follower movement, comment spikes, important profile metadata changes
high: new post detected, post removed, active campaign shift, landing page change

Once you do that, it becomes much easier to decide:

what goes to Slack immediately
what gets saved for a daily digest
what only belongs in the audit log

Without severity, every change competes for attention equally. That is how monitoring systems become unreadable.

Honest Alternatives

There are a few ways to approach this.

Raw object diff libraries

Great for debugging.

Usually too noisy for social monitoring.

Event sourcing everything

Powerful if you have a larger architecture and a reason to keep full history.

Overkill for many small tools.

Handwritten per-entity diff logic

This is still my favorite for social systems.

It is boring, explicit, and much easier to reason about than magic diff output.

Where SociaVault Fits

This is one of those layers where I want the upstream data source to be the boring part.

I use SociaVault for the public social data collection layer, then keep my own diff logic in application code.

That lets me work on the part that actually creates value: deciding what changed and why it matters.

That is the product problem. Collection is just a dependency.

Final Take

The hardest part of social monitoring is not polling.

It is meaning.

Your diff engine decides whether your product becomes a useful signal layer or a machine for generating trivia.

Normalize first. Use thresholds for numbers. Use text normalization for strings. Use set logic for collections. Add severity.

That combination took my monitoring systems from noisy to actually usable.

And if you want to spend more of your time on that diff logic instead of on collection plumbing, SociaVault is a good place to start.

webdev #monitoring #javascript #python #backend

How I'd Build a Landing Page Change Tracker for Competitor Ads

Olamide Olaniyan — Tue, 05 May 2026 19:19:00 +0000

Ad libraries tell you what the ad says.

The landing page tells you what the company is actually trying to make you do.

That difference matters.

I have seen teams obsess over competitor headlines while ignoring the part that changed first: the page behind the ad.

That is where you notice things like:

a new self-serve CTA
a new pricing angle
a comparison page replacing a generic feature page
a trial push replacing a demo flow
a new proof block added to support a fresh campaign

If you are tracking competitors seriously, ad monitoring without landing-page tracking is incomplete.

So this is the workflow I would build: use public ad data to discover the active landing pages, save structured snapshots of those pages, and diff the important parts over time.

Why Ad Pages Are Better Than General Site Crawling

You can crawl a competitor site broadly.

Sometimes that is useful.

But if your real goal is paid-intel and offer monitoring, ad-linked landing pages are much higher signal.

They tell you what the company is actively paying to promote.

That means the workflow is simpler and the insight is usually sharper.

The Three Things I Track First

I start with only a few fields.

page title
primary H1 or hero headline
primary CTA text
canonical URL

That is enough to detect most meaningful competitor landing-page changes.

You do not need a full DOM diff to start getting value.

JavaScript Version: Pull Ad URLs, Then Diff Key Page Elements

This version uses Facebook and LinkedIn ad data to collect landing page URLs, then fetches and compares a few key HTML elements.

const headers = { 'X-API-Key': process.env.SOCIAVAULT_API_KEY };

async function fetchJson(url) {
  const response = await fetch(url, { headers });
  if (!response.ok) {
    throw new Error(`Request failed with ${response.status}`);
  }
  return response.json();
}

async function fetchHtml(url) {
  const response = await fetch(url, {
    headers: {
      'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
    },
  });
  if (!response.ok) {
    throw new Error(`HTML fetch failed with ${response.status}`);
  }
  return response.text();
}

function extractBetween(html, regex) {
  const match = html.match(regex);
  return match ? match[1].replace(/\s+/g, ' ').trim() : null;
}

function parsePage(html, url) {
  return {
    url,
    title: extractBetween(html, /<title>(.*?)<\/title>/is),
    h1: extractBetween(html, /<h1[^>]*>(.*?)<\/h1>/is),
    cta: extractBetween(html, /<a[^>]*>(Start free|Book a demo|Talk to sales|Try free|Get started)<\/a>/is),
  };
}

function normalizeAds(items = []) {
  return (items || []).map(item => item.url || item.landingPageUrl || item.snapshot?.link_url).filter(Boolean);
}

function diffPage(previous, current) {
  const changes = [];

  for (const field of ['title', 'h1', 'cta']) {
    if ((previous?.[field] || null) !== (current?.[field] || null)) {
      changes.push({
        field,
        previous: previous?.[field] || null,
        current: current?.[field] || null,
      });
    }
  }

  return changes;
}

async function collectLandingPages(company) {
  const [facebookJson, linkedinJson] = await Promise.all([
    fetchJson(
      `https://api.sociavault.com/v1/scrape/facebook-ad-library/company-ads?companyName=${encodeURIComponent(company)}&status=ACTIVE&trim=true`
    ),
    fetchJson(
      `https://api.sociavault.com/v1/scrape/linkedin-ad-library/search?company=${encodeURIComponent(company)}`
    ),
  ]);

  const urls = new Set([
    ...normalizeAds(facebookJson.data),
    ...normalizeAds(linkedinJson.data),
  ]);

  const pages = [];
  for (const url of urls) {
    try {
      const html = await fetchHtml(url);
      pages.push(parsePage(html, url));
    } catch (error) {
      console.error(`Failed to fetch ${url}:`, error.message);
    }
  }

  return pages;
}

const previousPages = {
  'https://example.com/demo': {
    title: 'Book a Demo | Example',
    h1: 'See Example in action',
    cta: 'Book a demo',
  },
};

const currentPages = await collectLandingPages('HubSpot');

for (const page of currentPages) {
  const changes = diffPage(previousPages[page.url], page);
  if (changes.length > 0) {
    console.log(`\nChanges detected for ${page.url}`);
    console.log(changes);
  }
}

This is enough to build a daily tracker that tells you when an ad-linked page changes its core message.

And if you want the public ad layer feeding that workflow without managing it manually, SociaVault makes that part much easier.

Python Version: Good for Scheduled Tracking Jobs

Python works very well for this kind of scheduled page-diff script.

import os
import re
import requests


HEADERS = {'X-API-Key': os.environ['SOCIAVAULT_API_KEY']}


def fetch_json(url):
    response = requests.get(url, headers=HEADERS, timeout=30)
    response.raise_for_status()
    return response.json()


def fetch_html(url):
    response = requests.get(
        url,
        headers={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'},
        timeout=30,
    )
    response.raise_for_status()
    return response.text


def extract_between(html, pattern):
    match = re.search(pattern, html, re.IGNORECASE | re.DOTALL)
    return re.sub(r'\s+', ' ', match.group(1)).strip() if match else None


def parse_page(html, url):
    return {
        'url': url,
        'title': extract_between(html, r'<title>(.*?)</title>'),
        'h1': extract_between(html, r'<h1[^>]*>(.*?)</h1>'),
        'cta': extract_between(html, r'<a[^>]*>(Start free|Book a demo|Talk to sales|Try free|Get started)</a>'),
    }


def normalize_ads(items=None):
    items = items or []
    urls = []
    for item in items:
        url = item.get('url') or item.get('landingPageUrl') or item.get('snapshot', {}).get('link_url')
        if url:
            urls.append(url)
    return urls


def diff_page(previous, current):
    changes = []
    for field in ['title', 'h1', 'cta']:
        old_value = (previous or {}).get(field)
        new_value = current.get(field)
        if old_value != new_value:
            changes.append({'field': field, 'previous': old_value, 'current': new_value})
    return changes


def collect_landing_pages(company):
    facebook = fetch_json(
        f'https://api.sociavault.com/v1/scrape/facebook-ad-library/company-ads?companyName={company}&status=ACTIVE&trim=true'
    )
    linkedin = fetch_json(
        f'https://api.sociavault.com/v1/scrape/linkedin-ad-library/search?company={company}'
    )

    urls = set(normalize_ads(facebook.get('data')) + normalize_ads(linkedin.get('data')))
    pages = []

    for url in urls:
        try:
            html = fetch_html(url)
            pages.append(parse_page(html, url))
        except Exception as error:
            print(f'Failed to fetch {url}: {error}')

    return pages


previous_pages = {
    'https://example.com/demo': {
        'title': 'Book a Demo | Example',
        'h1': 'See Example in action',
        'cta': 'Book a demo',
    }
}

current_pages = collect_landing_pages('HubSpot')

for page in current_pages:
    changes = diff_page(previous_pages.get(page['url']), page)
    if changes:
        print(f'\nChanges detected for {page["url"]}')
        print(changes)

The Best Use Cases for This

This kind of tracker is especially useful for:

spotting pricing pivots earlier
seeing when a competitor moves from demo-led to self-serve
catching new proof points or trust elements in the hero section
identifying when a category starts leaning harder into comparison pages

That is much more actionable than just saving screenshots of ads.

Honest Alternatives

There are a few other ways to do this.

General site crawlers

Useful if you care about broad site changes.

Less focused if your real interest is what paid traffic is being sent to.

Visual diff tools

Great when design changes matter.

Often noisier and heavier than necessary if you only care about message shifts.

Manual weekly review

Fine for small lists.

Easy to forget, and very hard to compare consistently over time.

That is why I usually start with simple structured text diffs and only add visual diffing later if needed.

Final Take

If you want to understand what a competitor is actually pushing, look past the ad and track the page.

That is where the real shift often shows up first.

Use public ad data to find the current destination URLs. Save a few meaningful fields. Diff them over time. That alone gives you a much more useful competitor signal than casual ad-library browsing.

And if you want the ad-side data layer without wiring it all manually, SociaVault is a good place to start.

Then keep the rest simple and boring: fetch, extract, diff, report.

That is enough to make a landing-page change tracker genuinely useful.

webdev #monitoring #python #javascript #marketing

What Facebook Video Transcripts Reveal Before Sales Calls Do

Olamide Olaniyan — Mon, 04 May 2026 16:16:00 +0000

Most teams treat competitor videos like brand content.

They watch a reel, nod a little, maybe save the link, then move on.

I think that leaves a lot of value on the table.

Because once you stop watching those videos as content and start treating them as data, a more useful question shows up:

what are they saying repeatedly, and what does that tell us about the market right now?

That is where transcripts become interesting.

Not because transcripts are glamorous, but because they make short-form video searchable, classifiable, and diffable.

And once you can do that, you can spot changes in:

positioning
offers
objections
proof points
CTA strategy

Often before those changes become obvious in sales calls or broader market chatter.

This is the workflow I would build if I wanted Facebook reels and video posts to become a structured competitor-intelligence source instead of random tabs in a browser.

Why Transcripts Matter More Than Views

Views tell you reach.

Transcripts tell you messaging.

If I am monitoring competitors, messaging is usually more useful.

I want to know:

which promises keep showing up
which customer pains they repeat
what proof they lead with
whether the CTA changed this week
whether they are moving upmarket, downmarket, or more self-serve

That is what transcripts make easier to capture.

The Workflow I Like

The workflow is simple.

Collect public Facebook video or reel URLs
Pull the transcript for each one
Split transcript text into short statements
Classify those statements into buckets
Compare those buckets across time

You do not need a giant NLP stack to get useful output.

You just need enough structure to stop relying on memory.

JavaScript Version: Transcript Bucketing for Competitor Research

This starts with the Facebook transcript route and then turns transcript text into reusable messaging buckets.

const headers = {
  'X-API-Key': process.env.SOCIAVAULT_API_KEY,
};

async function fetchJson(url) {
  const response = await fetch(url, { headers });
  if (!response.ok) {
    throw new Error(`Request failed with ${response.status}`);
  }
  return response.json();
}

function extractTranscript(data = {}) {
  if (typeof data === 'string') return data;
  if (typeof data.transcript === 'string') return data.transcript;
  if (typeof data.text === 'string') return data.text;
  if (Array.isArray(data.captions)) {
    return data.captions.map(item => item.text || item.caption || '').join(' ');
  }
  return '';
}

function splitStatements(text) {
  return text
    .split(/[.!?\n]+/)
    .map(part => part.trim())
    .filter(Boolean)
    .filter(part => part.length > 20);
}

function classifyStatement(statement) {
  const value = statement.toLowerCase();

  if (/(struggling|frustrated|hard|difficult|problem|stuck|wasting time)/.test(value)) return 'pain';
  if (/(customers|teams|creators|brands|founders|agencies)/.test(value)) return 'audience';
  if (/(results|case study|revenue|saved|grew|increase|improved)/.test(value)) return 'proof';
  if (/(free trial|book a demo|get started|sign up|dm us|click the link)/.test(value)) return 'cta';
  if (/(price|pricing|plan|subscription|monthly|annual|cost)/.test(value)) return 'offer';
  return 'positioning';
}

async function analyzeTranscript(url) {
  const json = await fetchJson(
    `https://api.sociavault.com/v1/scrape/facebook/post/transcript?url=${encodeURIComponent(url)}`
  );

  const transcript = extractTranscript(json.data || {});
  const statements = splitStatements(transcript);
  const buckets = {
    pain: [],
    audience: [],
    proof: [],
    offer: [],
    cta: [],
    positioning: [],
  };

  for (const statement of statements) {
    buckets[classifyStatement(statement)].push(statement);
  }

  return {
    url,
    transcriptLength: transcript.length,
    bucketCounts: Object.fromEntries(Object.entries(buckets).map(([key, value]) => [key, value.length])),
    examples: Object.fromEntries(Object.entries(buckets).map(([key, value]) => [key, value.slice(0, 3)])),
  };
}

analyzeTranscript('https://www.facebook.com/reel/1535656380759655')
  .then(report => console.log(report))
  .catch(error => console.error(error));

This is already enough to make transcript review much more systematic.

Instead of saying "their content feels more salesy lately," I can actually show which buckets increased.

Python Version: Compare Multiple Videos Over Time

Once you want weekly or monthly tracking, Python is a comfortable way to batch it.

import os
import re
import requests
from collections import Counter


HEADERS = {'X-API-Key': os.environ['SOCIAVAULT_API_KEY']}


def fetch_json(url):
    response = requests.get(url, headers=HEADERS, timeout=30)
    response.raise_for_status()
    return response.json()


def extract_transcript(data):
    if isinstance(data, str):
        return data
    if isinstance(data, dict):
        if isinstance(data.get('transcript'), str):
            return data['transcript']
        if isinstance(data.get('text'), str):
            return data['text']
        if isinstance(data.get('captions'), list):
            return ' '.join((item.get('text') or item.get('caption') or '') for item in data['captions'])
    return ''


def split_statements(text):
    parts = re.split(r'[.!?\n]+', text)
    return [part.strip() for part in parts if part.strip() and len(part.strip()) > 20]


def classify_statement(statement):
    value = statement.lower()
    if re.search(r'(struggling|frustrated|hard|difficult|problem|stuck|wasting time)', value):
        return 'pain'
    if re.search(r'(customers|teams|creators|brands|founders|agencies)', value):
        return 'audience'
    if re.search(r'(results|case study|revenue|saved|grew|increase|improved)', value):
        return 'proof'
    if re.search(r'(free trial|book a demo|get started|dm us|click the link|sign up)', value):
        return 'cta'
    if re.search(r'(price|pricing|plan|subscription|monthly|annual|cost)', value):
        return 'offer'
    return 'positioning'


def analyze_transcript(url):
    json_data = fetch_json(
        f'https://api.sociavault.com/v1/scrape/facebook/post/transcript?url={url}'
    )
    transcript = extract_transcript(json_data.get('data') or {})
    statements = split_statements(transcript)

    buckets = Counter(classify_statement(statement) for statement in statements)
    examples = {}
    for bucket in ['pain', 'audience', 'proof', 'offer', 'cta', 'positioning']:
        examples[bucket] = [statement for statement in statements if classify_statement(statement) == bucket][:3]

    return {
        'url': url,
        'transcriptLength': len(transcript),
        'bucketCounts': dict(buckets),
        'examples': examples,
    }


urls = [
    'https://www.facebook.com/reel/1535656380759655',
    'https://www.facebook.com/reel/1535656380759656',
]

reports = [analyze_transcript(url) for url in urls]
print(reports)

That is enough to answer questions like:

did their CTA mix change?
are they leaning harder into proof recently?
are they talking about pricing more often?
did they shift from pain-led messaging to offer-led messaging?

That is useful market intelligence from public video content.

Where This Gets Especially Useful

I like transcript analysis most in four cases.

1. Competitor sales messaging

Watch the words they keep repeating.

2. Offer changes

Notice when more pricing, trial, or package language starts showing up.

3. Objection mining

Competitor videos often answer customer concerns directly.

4. Content planning

You can see which positioning angles the market is saturating.

That helps your own team avoid parroting everyone else.

Honest Alternatives

You can absolutely do this manually.

For a handful of videos, manual review is fine.

But once you are tracking multiple competitors across time, transcripts are what make the workflow scalable.

You can also push this further with embeddings or LLM summarization later. I would just avoid starting there.

The basic transcript bucketing layer is already useful and much easier to trust.

Final Take

Competitor videos are not just content samples. They are messaging assets hiding in plain sight.

Transcripts turn them into something you can actually analyze over time.

If you want the public transcript layer without building the collection plumbing from scratch, SociaVault is a good place to start.

Then do the part that actually matters: classify what they keep saying, compare it across time, and use that signal before the market feels obvious.

That is usually where the edge is.

webdev #python #javascript #marketing #competitiveanalysis