DEV Community: Olamide Olaniyan

What YouTube Comment Replies Reveal Before User Interviews Do

Olamide Olaniyan — Sat, 09 May 2026 07:20:00 +0000

Most product teams stop at top-level comments.

That is usually where the obvious reactions live.

Replies are where people explain themselves.

That is a much better place to look if you care about:

objections
feature requests
product confusion
pricing resistance
onboarding friction
competitor comparisons

I still like user interviews. I still run them. I would never tell you YouTube comments should replace talking to real customers.

But if you are trying to find language, recurring problems, and messy real-world buyer questions before you book ten calls, YouTube comment replies are a very underused source.

This post shows you how I use them for product research, how to pull them with JavaScript and Python, and when this approach is smarter than interviews, surveys, or the official API.

Why Replies Matter More Than Top-Level Comments

Top-level comments are often performative.

People praise the creator. Joke about the video. Drop a quick take. Argue with the premise.

Replies are different.

Replies are where someone says:

"I tried this and got stuck at the migration step"
"This sounds good until you have to bring in a team"
"We switched away from this because the pricing got weird"
"Does this work if you need approval flows too?"

That is product language.

Not polished landing-page language. Not survey language. Not customer-success-call language.

Real language.

That is why I keep coming back to replies.

The Research Question Comes First

Do not start by scraping thousands of replies and hoping insight appears.

Start with a question.

Examples:

What objections keep showing up around this category?
What edge cases do people care about most?
What makes buyers hesitate?
What pain points do competitor users describe in their own words?

If you have that question first, the reply threads become much easier to read.

JavaScript Version: Pull Comments, Then Pull Replies

With YouTube, the workflow is straightforward.

First fetch top-level comments for a video. Then look for comments that include a repliesContinuationToken. That token is the key to the reply thread.

const headers = {
  'X-API-Key': process.env.SOCIAVAULT_API_KEY,
};

async function fetchJson(url) {
  const response = await fetch(url, { headers });
  if (!response.ok) {
    throw new Error(`Request failed with ${response.status}`);
  }
  return response.json();
}

async function getCommentThreads(videoUrl) {
  const commentsJson = await fetchJson(
    `https://api.sociavault.com/v1/scrape/youtube/video/comments?url=${encodeURIComponent(videoUrl)}&order=top`
  );

  const comments = Object.values(commentsJson.data?.comments || {});
  const threads = [];

  for (const comment of comments) {
    const token = comment.repliesContinuationToken;
    if (!token) continue;

    const repliesJson = await fetchJson(
      `https://api.sociavault.com/v1/scrape/youtube/video/comment-replies?continuationToken=${encodeURIComponent(token)}`
    );

    threads.push({
      topLevelComment: comment.content || comment.text,
      replies: repliesJson.data?.comments || repliesJson.data || [],
    });
  }

  return threads;
}

function classifyReply(text) {
  const value = (text || '').toLowerCase();

  if (/(too expensive|pricing|cost|price)/.test(value)) return 'pricing';
  if (/(integrat|api|sync|connect)/.test(value)) return 'integration';
  if (/(confus|hard|complicated|setup)/.test(value)) return 'onboarding';
  if (/(competitor|alternative|instead of|switched from)/.test(value)) return 'competitor';
  return 'other';
}

async function researchVideo(videoUrl) {
  const threads = await getCommentThreads(videoUrl);
  const summary = {};

  for (const thread of threads) {
    for (const reply of thread.replies) {
      const text = reply.content || reply.text || '';
      const label = classifyReply(text);
      summary[label] = summary[label] || [];
      summary[label].push(text);
    }
  }

  return summary;
}

researchVideo('https://www.youtube.com/watch?v=dQw4w9WgXcQ')
  .then(summary => console.log(summary))
  .catch(error => console.error(error));

That already gives you something useful: a rough thematic breakdown of reply language for one video.

Do that across five or ten relevant videos and patterns start showing up fast.

If you want a simpler way to get the public YouTube data without managing the collection layer, SociaVault is what I would use here.

Python Version: Same Workflow, Easy to Batch

Python is great for this kind of batch research workflow.

import os
import requests


HEADERS = {'X-API-Key': os.environ['SOCIAVAULT_API_KEY']}


def fetch_json(url):
    response = requests.get(url, headers=HEADERS, timeout=30)
    response.raise_for_status()
    return response.json()


def get_comment_threads(video_url):
    comments_json = fetch_json(
        f'https://api.sociavault.com/v1/scrape/youtube/video/comments?url={video_url}&order=top'
    )

    comments = list((comments_json.get('data') or {}).get('comments', {}).values())
    threads = []

    for comment in comments:
        token = comment.get('repliesContinuationToken')
        if not token:
            continue

        replies_json = fetch_json(
            f'https://api.sociavault.com/v1/scrape/youtube/video/comment-replies?continuationToken={token}'
        )

        threads.append({
            'topLevelComment': comment.get('content') or comment.get('text'),
            'replies': replies_json.get('data', {}).get('comments') or replies_json.get('data') or [],
        })

    return threads


def classify_reply(text):
    value = (text or '').lower()

    if any(term in value for term in ['too expensive', 'pricing', 'cost', 'price']):
        return 'pricing'
    if any(term in value for term in ['integrat', 'api', 'sync', 'connect']):
        return 'integration'
    if any(term in value for term in ['confus', 'hard', 'complicated', 'setup']):
        return 'onboarding'
    if any(term in value for term in ['competitor', 'alternative', 'instead of', 'switched from']):
        return 'competitor'
    return 'other'


def research_video(video_url):
    threads = get_comment_threads(video_url)
    summary = {}

    for thread in threads:
        for reply in thread['replies']:
            text = reply.get('content') or reply.get('text') or ''
            label = classify_reply(text)
            summary.setdefault(label, []).append(text)

    return summary


summary = research_video('https://www.youtube.com/watch?v=dQw4w9WgXcQ')
for label, examples in summary.items():
    print(f'\n## {label.upper()}')
    for example in examples[:5]:
        print('-', example)

The nice thing about the Python version is that it is very easy to turn into a weekly batch job that analyzes several videos in one pass.

What To Do With the Reply Data

This is the part people often skip.

Do not just collect replies.

Turn them into something reusable.

My usual workflow is:

pick 5-10 relevant videos in the category
pull top-level comments
fetch reply threads where discussion actually happened
tag replies by theme
save exact phrases, not just summaries

That fifth step matters a lot.

You want the real wording.

If ten people say some version of:

looks useful, but this feels too complicated for a small team

that phrase is often more valuable than a neat internal note like "SMB onboarding concerns exist."

The messy language is usually the useful language.

Where This Is Better Than Interviews

This is better than user interviews when:

you need fast raw language
you are early and need hypothesis fuel
you want to study a competitor category before reaching out to users
you want public objections at scale

It is especially useful before writing:

landing page copy
pricing FAQs
comparison pages
ad hooks
onboarding messaging

Where Interviews Still Win

This is where I want to be very clear.

YouTube replies are not a replacement for talking to users.

Interviews are still much better when you need:

follow-up questions
segment-specific nuance
willingness-to-pay signals
workflow details that people do not write publicly
actual product validation

Think of reply analysis as the thing that helps you ask better interview questions.

Not the thing that replaces interviews.

Honest Alternatives

There are three realistic options here.

The official YouTube API

Good if you are already comfortable inside Google's ecosystem and only need YouTube.

Manual reading

Good for quick one-off research.

Bad if you want something repeatable or cross-video.

Public data workflow with SociaVault

Best when you want a simpler API flow and expect this research to become part of a larger stack.

That is usually where I land, because I rarely want a one-platform-only solution for very long.

Final Take

Top-level comments tell you the reaction.

Replies tell you the reasoning.

That is why I keep using them for product research, competitor analysis, copywriting, and objection mining.

If you want a practical way to turn YouTube reply threads into reusable research, SociaVault gives you the public data layer so you can spend your time on analysis instead of collection.

Start small. Pick five videos. Tag the replies. Save the exact phrases.

You will usually get better language out of that exercise than out of a week of guessing.

webdev #youtube #productivity #python #javascript

How I'd Vet Creators for Whitelisting and Paid Media Access

Olamide Olaniyan — Fri, 08 May 2026 21:19:00 +0000

Whitelisting is where creator vetting gets more serious.

A normal sponsorship asks, "Should we work with this person?"

Whitelisting asks a more expensive question:

should we put paid spend behind this creator's identity, content style, and public trust?

That is a higher bar.

And a lot of teams still evaluate it with roughly the same checklist they use for ordinary creator partnerships.

I would not.

Because once paid media is involved, weak judgment gets amplified.

The wrong creator is not just a soft underperformer anymore. They become a scaled distribution problem.

This is the review workflow I would build before approving any creator for whitelisting or broader paid media access.

The Core Difference

For a standard partnership, decent content fit might be enough.

For whitelisting, I want stronger answers to four things:

does the creator feel credible on camera?
does their recent content look stable and brand-safe?
does their audience engagement look believable enough?
can I imagine running spend through this identity without regret?

That last question matters more than people admit.

The Risk Areas I Care About

The first-pass vetting layer I want includes:

1. Creative consistency

If the recent content is chaotic, stale, or all over the place, that matters.

2. Brand safety

Not just obvious harmful content. Also erratic tone, sloppy claims, or recurring controversy patterns.

3. Audience reliability

Paid media does not fix weak trust.

4. Operational consistency

If posting activity is inconsistent or the content mix shifts wildly, that is a risk too.

JavaScript Version: First-Pass Whitelisting Readiness Check

This uses recent TikTok videos as the public signal layer, then scores readiness conservatively.

const headers = {
  'X-API-Key': process.env.SOCIAVAULT_API_KEY,
};

async function fetchJson(url) {
  const response = await fetch(url, { headers });
  if (!response.ok) {
    throw new Error(`Request failed with ${response.status}`);
  }
  return response.json();
}

function average(values) {
  if (!values.length) return 0;
  return values.reduce((sum, value) => sum + value, 0) / values.length;
}

function containsRiskyLanguage(text = '') {
  return /(hate|scam|guaranteed income|get rich quick|offensive|explicit)/i.test(text);
}

function meaningfulCaption(text = '') {
  return (text || '').trim().length >= 12;
}

function buildDecision(score) {
  if (score >= 75) return 'approve';
  if (score >= 55) return 'caution';
  return 'decline';
}

async function assessWhitelisting(handle) {
  const [profileJson, videosJson, followersJson] = await Promise.all([
    fetchJson(`https://api.sociavault.com/v1/scrape/tiktok/profile?handle=${encodeURIComponent(handle)}`),
    fetchJson(`https://api.sociavault.com/v1/scrape/tiktok/videos?handle=${encodeURIComponent(handle)}&amount=12&sort_by=latest&trim=true`),
    fetchJson(`https://api.sociavault.com/v1/scrape/tiktok/followers?handle=${encodeURIComponent(handle)}&trim=true`),
  ]);

  const profile = profileJson.data || {};
  const videos = videosJson.data || [];
  const followers = followersJson.data || [];

  const followerCount = profile.stats?.followerCount || profile.followerCount || 0;
  const engagements = videos.map(video => (video.stats?.diggCount || 0) + (video.stats?.commentCount || 0));
  const engagementRate = followerCount > 0 ? (average(engagements) / followerCount) * 100 : 0;

  const brandSafetyFlags = videos.filter(video => containsRiskyLanguage(video.desc || '')).length;
  const weakCaptionShare = videos.length
    ? videos.filter(video => !meaningfulCaption(video.desc || '')).length / videos.length
    : 0;

  const suspiciousFollowerShare = followers.length
    ? followers.filter(follower => {
        const username = follower.username || follower.unique_id || '';
        return /\d{5,}/.test(username);
      }).length / followers.length
    : 0;

  let score = 100;
  if (engagementRate < 1) score -= 20;
  if (brandSafetyFlags > 0) score -= 25;
  if (weakCaptionShare > 0.5) score -= 15;
  if (suspiciousFollowerShare > 0.3) score -= 20;
  if (videos.length < 8) score -= 10;

  return {
    handle,
    followerCount,
    engagementRate: Number(engagementRate.toFixed(2)),
    brandSafetyFlags,
    weakCaptionShare: Number(weakCaptionShare.toFixed(2)),
    suspiciousFollowerShare: Number(suspiciousFollowerShare.toFixed(2)),
    recentVideoCount: videos.length,
    score,
    decision: buildDecision(score),
  };
}

assessWhitelisting('creator_handle')
  .then(report => console.log(report))
  .catch(error => console.error(error));

This is not a legal approval system.

It is a strong first-pass screen that stops weak candidates from getting too far.

Python Version: Batch Whitelisting Reviews for Creator Lists

If I were screening a campaign list, I would probably use Python.

import os
import re
import requests


HEADERS = {'X-API-Key': os.environ['SOCIAVAULT_API_KEY']}


def fetch_json(url):
    response = requests.get(url, headers=HEADERS, timeout=30)
    response.raise_for_status()
    return response.json()


def average(values):
    return sum(values) / len(values) if values else 0


def contains_risky_language(text=''):
    return bool(re.search(r'(hate|scam|guaranteed income|get rich quick|offensive|explicit)', text, re.IGNORECASE))


def meaningful_caption(text=''):
    return len((text or '').strip()) >= 12


def build_decision(score):
    if score >= 75:
        return 'approve'
    if score >= 55:
        return 'caution'
    return 'decline'


def assess_whitelisting(handle):
    profile_json = fetch_json(f'https://api.sociavault.com/v1/scrape/tiktok/profile?handle={handle}')
    videos_json = fetch_json(f'https://api.sociavault.com/v1/scrape/tiktok/videos?handle={handle}&amount=12&sort_by=latest&trim=true')
    followers_json = fetch_json(f'https://api.sociavault.com/v1/scrape/tiktok/followers?handle={handle}&trim=true')

    profile = profile_json.get('data') or {}
    videos = videos_json.get('data') or []
    followers = followers_json.get('data') or []

    follower_count = profile.get('stats', {}).get('followerCount') or profile.get('followerCount') or 0
    engagements = [(video.get('stats', {}).get('diggCount', 0) + video.get('stats', {}).get('commentCount', 0)) for video in videos]
    engagement_rate = (average(engagements) / follower_count * 100) if follower_count else 0

    brand_safety_flags = sum(1 for video in videos if contains_risky_language(video.get('desc') or ''))
    weak_caption_share = (sum(1 for video in videos if not meaningful_caption(video.get('desc') or '')) / len(videos)) if videos else 0
    suspicious_follower_share = (
        sum(1 for follower in followers if re.search(r'\d{5,}', follower.get('username') or follower.get('unique_id') or '')) / len(followers)
        if followers else 0
    )

    score = 100
    if engagement_rate < 1:
        score -= 20
    if brand_safety_flags > 0:
        score -= 25
    if weak_caption_share > 0.5:
        score -= 15
    if suspicious_follower_share > 0.3:
        score -= 20
    if len(videos) < 8:
        score -= 10

    return {
        'handle': handle,
        'followerCount': follower_count,
        'engagementRate': round(engagement_rate, 2),
        'brandSafetyFlags': brand_safety_flags,
        'weakCaptionShare': round(weak_caption_share, 2),
        'suspiciousFollowerShare': round(suspicious_follower_share, 2),
        'recentVideoCount': len(videos),
        'score': score,
        'decision': build_decision(score),
    }


handles = ['creator_one', 'creator_two', 'creator_three']
reports = [assess_whitelisting(handle) for handle in handles]
print(reports)

What This Workflow Does Not Replace

This part matters.

Whitelisting approval should still include manual review for:

contract scope
usage rights
platform access details
final brand-safety judgment
campaign-specific creative fit

The point of the workflow is not to automate those decisions away.

The point is to stop weak candidates earlier.

Honest Alternatives

You can run this process entirely manually. Plenty of teams still do.

But if you are screening real creator volume, manual whitelisting review becomes inconsistent very quickly.

You can also buy a full creator platform, which may be the right move for larger teams.

I like a lighter setup when the actual value is in the review logic itself. In that model, SociaVault gives me the public social data layer and I keep the approval rules in my own workflow.

That is a cleaner separation.

Final Take

Whitelisting deserves a stricter review standard than a normal creator collaboration.

That is the main idea.

If you are about to put spend behind a creator's identity, you want more than a nice profile and a decent average view count.

You want evidence that the content is stable, the engagement is believable, the recent output is usable, and the risk level is acceptable.

If you want to build that pre-approval layer without wiring the public data collection stack from scratch, SociaVault is a good place to start.

Then keep the review honest and conservative.

That is usually the difference between a scalable paid-media workflow and a very expensive lesson.

webdev #python #javascript #marketing #creatoreconomy

I Tried to Normalize 8 Social Platforms Into One JSON Schema. Here's What Broke

Olamide Olaniyan — Thu, 07 May 2026 19:19:00 +0000

I thought this was going to be easy.

Take data from TikTok, Instagram, YouTube, Reddit, LinkedIn, Threads, X, and Facebook. Map it into one clean shape. Ship it. Move on.

That lasted about a day.

By day two, I had already learned the first ugly truth of social data engineering:

the hard part is not collecting the data. The hard part is deciding what the same thing even means across platforms.

"Likes" are not always likes. "Views" are not always views. Some platforms expose shares publicly, some do not. One platform gives you createTime, another gives you ISO timestamps, another gives you nested objects with three different possible IDs depending on the endpoint.

If you're building a social dashboard, creator analytics tool, moderation workflow, or competitor monitoring system, you hit this wall fast.

So this post is the version I wish I had read earlier: what broke, what schema I kept, what I stopped trying to normalize, and how I now build a social media JSON schema without lying to myself.

The Mistake I Made First

My first schema looked clean.

{
  "id": "123",
  "platform": "tiktok",
  "author": "creator_handle",
  "text": "post text",
  "likes": 100,
  "comments": 12,
  "views": 5000,
  "shares": 4,
  "published_at": "2026-04-24T12:00:00Z"
}

That shape is attractive because it feels universal.

It is also incomplete in exactly the ways that cause expensive bugs later.

Here is what broke almost immediately:

TikTok and Instagram expose author and stats differently.
YouTube comments do not behave like short-form video posts.
Reddit has score, comments, and post metadata that do not map cleanly to "likes/views/shares."
LinkedIn often has less public engagement detail than people expect.
Some endpoints return counts as nested stats objects. Others flatten them.
URLs and canonical IDs vary wildly by platform.
Missing values are common, and missing does not mean zero.

That last point matters a lot.

If a platform does not expose a public field, setting it to 0 is false precision. You are making downstream analysis worse while pretending you are making it simpler.

The Rule That Saved Me

I stopped trying to force every platform into the exact same semantic model.

Instead, I switched to this rule:

Normalize the envelope. Preserve the raw meaning.

That means every object should share a predictable wrapper, but platform-specific nuance should stay visible.

The schema I ended up liking looks more like this:

{
  "platform": "tiktok",
  "entity_type": "post",
  "external_id": "735902002991",
  "canonical_url": "https://www.tiktok.com/@creator/video/735902002991",
  "author": {
    "handle": "creator",
    "display_name": "Creator Name"
  },
  "content": {
    "text": "caption text",
    "media_type": "video"
  },
  "metrics": {
    "likes": 1200,
    "comments": 87,
    "shares": 41,
    "views": 54000
  },
  "published_at": "2026-04-24T12:00:00Z",
  "availability": {
    "views": "public",
    "shares": "public",
    "saves": "not_available"
  },
  "raw": {}
}

Notice what changed:

metrics is grouped instead of flattened.
availability tells downstream code whether a field is real, missing, or unavailable.
raw is preserved for debugging, reprocessing, and future schema changes.

This is much less elegant than the first version.

It is also much more survivable.

What I Normalize Now

These are the fields I consider worth normalizing across platforms:

platform
entity_type
external_id
canonical_url
author.handle
author.display_name
content.text
content.media_type
metrics.likes
metrics.comments
metrics.views
metrics.shares
published_at

That is enough to make cross-platform dashboards, alerts, ranking jobs, and exports practical.

But I do not pretend every field exists everywhere.

JavaScript Version: A Normalizer That Stays Honest

This example is intentionally boring. That is a good sign.

If your normalization layer is too clever, it becomes unmaintainable fast.

function pickFirst(...values) {
  return values.find(value => value !== undefined && value !== null && value !== '');
}

function toNumber(value) {
  const number = Number(value);
  return Number.isFinite(number) ? number : null;
}

function toIsoDate(value) {
  if (!value) return null;

  if (typeof value === 'number') {
    return new Date(value * 1000).toISOString();
  }

  const parsed = new Date(value);
  return Number.isNaN(parsed.getTime()) ? null : parsed.toISOString();
}

function normalizePost(platform, raw) {
  const base = {
    platform,
    entity_type: 'post',
    external_id: null,
    canonical_url: null,
    author: {
      handle: null,
      display_name: null,
    },
    content: {
      text: null,
      media_type: null,
    },
    metrics: {
      likes: null,
      comments: null,
      views: null,
      shares: null,
    },
    published_at: null,
    availability: {
      likes: 'unknown',
      comments: 'unknown',
      views: 'unknown',
      shares: 'unknown',
    },
    raw,
  };

  if (platform === 'tiktok') {
    return {
      ...base,
      external_id: pickFirst(raw.id, raw.aweme_id),
      canonical_url: pickFirst(raw.share_url, raw.url),
      author: {
        handle: pickFirst(raw.author?.uniqueId, raw.author?.handle),
        display_name: pickFirst(raw.author?.nickname, raw.author?.displayName),
      },
      content: {
        text: pickFirst(raw.desc, raw.caption),
        media_type: 'video',
      },
      metrics: {
        likes: toNumber(pickFirst(raw.stats?.diggCount, raw.like_count)),
        comments: toNumber(pickFirst(raw.stats?.commentCount, raw.comment_count)),
        views: toNumber(pickFirst(raw.stats?.playCount, raw.view_count)),
        shares: toNumber(pickFirst(raw.stats?.shareCount, raw.share_count)),
      },
      published_at: toIsoDate(pickFirst(raw.createTime, raw.created_at)),
      availability: {
        likes: 'public',
        comments: 'public',
        views: 'public',
        shares: 'public',
      },
    };
  }

  if (platform === 'youtube_comment') {
    return {
      ...base,
      entity_type: 'comment',
      external_id: pickFirst(raw.commentId, raw.id),
      canonical_url: pickFirst(raw.url),
      author: {
        handle: pickFirst(raw.author?.channelHandle, raw.author?.name),
        display_name: pickFirst(raw.author?.name, raw.authorDisplayName),
      },
      content: {
        text: pickFirst(raw.content, raw.text),
        media_type: 'text',
      },
      metrics: {
        likes: toNumber(pickFirst(raw.likes, raw.likeCount)),
        comments: toNumber(pickFirst(raw.replyCount, raw.repliesCount)),
        views: null,
        shares: null,
      },
      published_at: toIsoDate(pickFirst(raw.publishedAt, raw.timestamp)),
      availability: {
        likes: 'public',
        comments: 'public',
        views: 'not_available',
        shares: 'not_available',
      },
    };
  }

  if (platform === 'reddit') {
    return {
      ...base,
      external_id: pickFirst(raw.id, raw.post_id),
      canonical_url: pickFirst(raw.permalink, raw.url),
      author: {
        handle: pickFirst(raw.author, raw.author_name),
        display_name: pickFirst(raw.author, raw.author_name),
      },
      content: {
        text: pickFirst(raw.selftext, raw.title),
        media_type: raw.is_video ? 'video' : 'text',
      },
      metrics: {
        likes: toNumber(pickFirst(raw.score, raw.upvotes)),
        comments: toNumber(pickFirst(raw.num_comments, raw.comment_count)),
        views: null,
        shares: null,
      },
      published_at: toIsoDate(pickFirst(raw.created_utc, raw.created_at)),
      availability: {
        likes: 'public_proxy',
        comments: 'public',
        views: 'not_available',
        shares: 'not_available',
      },
    };
  }

  throw new Error(`Unsupported platform: ${platform}`);
}

const tiktokExample = {
  id: '735902002991',
  desc: 'How I batch content research in 15 minutes',
  createTime: 1713936000,
  author: { uniqueId: 'creator_handle', nickname: 'Creator Name' },
  stats: { diggCount: 1200, commentCount: 87, playCount: 54000, shareCount: 41 },
  share_url: 'https://www.tiktok.com/@creator_handle/video/735902002991',
};

console.log(normalizePost('tiktok', tiktokExample));

The important part is not the exact shape above.

The important part is that unsupported or unavailable fields stay explicit.

That makes your downstream analytics much easier to trust.

Python Version: Same Strategy, Different Runtime

If your data jobs live in Python, the same idea applies.

from datetime import datetime, timezone


def pick_first(*values):
    for value in values:
        if value not in (None, '', []):
            return value
    return None


def to_number(value):
    try:
        return int(value)
    except (TypeError, ValueError):
        return None


def to_iso_date(value):
    if value is None:
        return None

    if isinstance(value, (int, float)):
        return datetime.fromtimestamp(value, tz=timezone.utc).isoformat()

    try:
        return datetime.fromisoformat(str(value).replace('Z', '+00:00')).isoformat()
    except ValueError:
        return None


def normalize_post(platform, raw):
    base = {
        'platform': platform,
        'entity_type': 'post',
        'external_id': None,
        'canonical_url': None,
        'author': {
            'handle': None,
            'display_name': None,
        },
        'content': {
            'text': None,
            'media_type': None,
        },
        'metrics': {
            'likes': None,
            'comments': None,
            'views': None,
            'shares': None,
        },
        'published_at': None,
        'availability': {
            'likes': 'unknown',
            'comments': 'unknown',
            'views': 'unknown',
            'shares': 'unknown',
        },
        'raw': raw,
    }

    if platform == 'tiktok':
        return {
            **base,
            'external_id': pick_first(raw.get('id'), raw.get('aweme_id')),
            'canonical_url': pick_first(raw.get('share_url'), raw.get('url')),
            'author': {
                'handle': pick_first(raw.get('author', {}).get('uniqueId'), raw.get('author', {}).get('handle')),
                'display_name': pick_first(raw.get('author', {}).get('nickname'), raw.get('author', {}).get('displayName')),
            },
            'content': {
                'text': pick_first(raw.get('desc'), raw.get('caption')),
                'media_type': 'video',
            },
            'metrics': {
                'likes': to_number(pick_first(raw.get('stats', {}).get('diggCount'), raw.get('like_count'))),
                'comments': to_number(pick_first(raw.get('stats', {}).get('commentCount'), raw.get('comment_count'))),
                'views': to_number(pick_first(raw.get('stats', {}).get('playCount'), raw.get('view_count'))),
                'shares': to_number(pick_first(raw.get('stats', {}).get('shareCount'), raw.get('share_count'))),
            },
            'published_at': to_iso_date(pick_first(raw.get('createTime'), raw.get('created_at'))),
            'availability': {
                'likes': 'public',
                'comments': 'public',
                'views': 'public',
                'shares': 'public',
            },
        }

    if platform == 'youtube_comment':
        return {
            **base,
            'entity_type': 'comment',
            'external_id': pick_first(raw.get('commentId'), raw.get('id')),
            'canonical_url': raw.get('url'),
            'author': {
                'handle': pick_first(raw.get('author', {}).get('channelHandle'), raw.get('author', {}).get('name')),
                'display_name': pick_first(raw.get('author', {}).get('name'), raw.get('authorDisplayName')),
            },
            'content': {
                'text': pick_first(raw.get('content'), raw.get('text')),
                'media_type': 'text',
            },
            'metrics': {
                'likes': to_number(pick_first(raw.get('likes'), raw.get('likeCount'))),
                'comments': to_number(pick_first(raw.get('replyCount'), raw.get('repliesCount'))),
                'views': None,
                'shares': None,
            },
            'published_at': to_iso_date(pick_first(raw.get('publishedAt'), raw.get('timestamp'))),
            'availability': {
                'likes': 'public',
                'comments': 'public',
                'views': 'not_available',
                'shares': 'not_available',
            },
        }

    if platform == 'reddit':
        return {
            **base,
            'external_id': pick_first(raw.get('id'), raw.get('post_id')),
            'canonical_url': pick_first(raw.get('permalink'), raw.get('url')),
            'author': {
                'handle': pick_first(raw.get('author'), raw.get('author_name')),
                'display_name': pick_first(raw.get('author'), raw.get('author_name')),
            },
            'content': {
                'text': pick_first(raw.get('selftext'), raw.get('title')),
                'media_type': 'video' if raw.get('is_video') else 'text',
            },
            'metrics': {
                'likes': to_number(pick_first(raw.get('score'), raw.get('upvotes'))),
                'comments': to_number(pick_first(raw.get('num_comments'), raw.get('comment_count'))),
                'views': None,
                'shares': None,
            },
            'published_at': to_iso_date(pick_first(raw.get('created_utc'), raw.get('created_at'))),
            'availability': {
                'likes': 'public_proxy',
                'comments': 'public',
                'views': 'not_available',
                'shares': 'not_available',
            },
        }

    raise ValueError(f'Unsupported platform: {platform}')


tiktok_example = {
    'id': '735902002991',
    'desc': 'How I batch content research in 15 minutes',
    'createTime': 1713936000,
    'author': {'uniqueId': 'creator_handle', 'nickname': 'Creator Name'},
    'stats': {'diggCount': 1200, 'commentCount': 87, 'playCount': 54000, 'shareCount': 41},
    'share_url': 'https://www.tiktok.com/@creator_handle/video/735902002991',
}

print(normalize_post('tiktok', tiktok_example))

What I Stopped Trying to Normalize

This part saved me more time than any helper function.

There are fields I no longer force into a universal schema unless the use case truly needs them:

saves/bookmarks
watch time
audience demographics
thread depth nuances
platform-specific moderation states
estimated reach proxies

Why?

Because those fields mean different things, appear inconsistently, or are missing often enough that the abstraction becomes misleading.

If a product feature really depends on one of those platform-specific concepts, I expose that field under a platform namespace instead.

That keeps the shared schema clean and the specialized data honest.

Honest Alternatives

There are three valid approaches here.

1. Normalize everything into one schema

Good for dashboards and cross-platform ranking.

Bad if you start pretending unavailable fields are universal.

2. Keep completely separate per-platform models

Good for correctness.

Bad for product teams that want a single report, feed, or analytics layer.

3. Keep a thin shared schema plus raw JSON

This is the one I use most often.

It gives you enough consistency for application logic without throwing away nuance.

If you are early, start there.

Where SociaVault Fits

This is exactly where a data layer like SociaVault helps.

I do not want to spend my week maintaining eight different scraping stacks just to get to the actual product problem, which is normalization, ranking, alerting, and reporting.

So my usual split is:

use SociaVault for the public social data collection layer
normalize into my own schema
keep the raw payload beside the normalized object

That keeps the engineering work focused on product logic instead of collection plumbing.

Final Take

Trying to build a universal social media JSON schema taught me something useful: the goal is not to erase platform differences.

The goal is to make those differences manageable.

Normalize the envelope. Preserve the raw payload. Mark unavailable fields honestly. Avoid fake precision.

If you are building social dashboards, creator analytics, competitor tracking, or moderation tools, that approach will hold up much better than the "one perfect schema" fantasy.

And if you want to skip the collection layer and spend your time on the normalization layer instead, start with SociaVault.

webdev #api #json #datascience #javascript

What I Learned Building a Diff Engine for Social Profiles, Posts, and Follower Counts

Olamide Olaniyan — Wed, 06 May 2026 22:17:00 +0000

The first version of my social monitoring system was technically correct and practically useless.

Every poll produced changes.

Follower count up by 3.

Bio spacing changed.

A post metric moved slightly.

A timestamp field came back in a different format.

The system was full of "change," and almost none of it mattered.

That was when I realized a social monitoring product does not really need polling logic first.

It needs a good diff engine.

Because if you cannot decide what changed meaningfully, the rest of the stack becomes noise generation.

So this is the diff model I use now for profiles, posts, and follower counts: what I compare, what I ignore, how I implement it in JavaScript and Python, and where a public social data layer like SociaVault makes the whole workflow much easier.

What a Good Diff Engine Actually Does

I think of a diff engine as having three jobs:

normalize the old and new state into comparable shapes
detect meaningful differences
classify those differences by importance

That third part is the one most implementations skip.

A change is not automatically an event.

That distinction is the difference between useful monitoring and alert fatigue.

The Three Objects I Compare Most Often

In social systems, I usually diff three categories.

1. Profiles

follower count
following count
bio
display name
profile image URL
post count or video count

2. Posts

caption or body text
likes
comments
views
pinned status if available
availability or deletion state

3. Collections

newest post IDs
newest comment IDs
active ad signatures
landing page URL sets

The rules are different for each one.

That matters.

My Threshold Rule

This one changed the quality of my alerts immediately:

numeric fields need thresholds. text fields need normalization. collection fields need set comparison.

If you diff all three types the same way, you get garbage.

For example:

follower changes need percentage or absolute thresholds
bios should be trimmed and whitespace-normalized first
post IDs should be compared as sets
metric deltas should often be classified, not just detected

That is the whole trick.

JavaScript Version: Diffing Social State Without Lying

This is the kind of diff function I like in Node services.

function normalizeText(value) {
  return (value || '').replace(/\s+/g, ' ').trim();
}

function diffNumber(previous, current, options = {}) {
  const { minAbsolute = 0, minPercent = 0 } = options;
  const prev = Number(previous || 0);
  const curr = Number(current || 0);
  const delta = curr - prev;
  const percent = prev > 0 ? Math.abs(delta / prev) * 100 : 0;

  const changed = Math.abs(delta) >= minAbsolute || percent >= minPercent;

  return {
    changed,
    previous: prev,
    current: curr,
    delta,
    percent: Number(percent.toFixed(2)),
  };
}

function diffText(previous, current) {
  const prev = normalizeText(previous);
  const curr = normalizeText(current);

  return {
    changed: prev !== curr,
    previous: prev,
    current: curr,
  };
}

function diffSet(previous = [], current = []) {
  const prev = new Set(previous);
  const curr = new Set(current);

  const added = [...curr].filter(item => !prev.has(item));
  const removed = [...prev].filter(item => !curr.has(item));

  return {
    changed: added.length > 0 || removed.length > 0,
    added,
    removed,
  };
}

function diffProfile(previous, current) {
  const followers = diffNumber(previous.followers, current.followers, {
    minAbsolute: 100,
    minPercent: 5,
  });

  const posts = diffNumber(previous.posts, current.posts, {
    minAbsolute: 1,
  });

  const bio = diffText(previous.bio, current.bio);
  const displayName = diffText(previous.displayName, current.displayName);

  const changes = [];

  if (followers.changed) {
    changes.push({ type: 'follower_change', ...followers, severity: 'medium' });
  }

  if (posts.changed && posts.delta > 0) {
    changes.push({ type: 'new_post_count', ...posts, severity: 'high' });
  }

  if (bio.changed) {
    changes.push({ type: 'bio_updated', ...bio, severity: 'low' });
  }

  if (displayName.changed) {
    changes.push({ type: 'display_name_updated', ...displayName, severity: 'low' });
  }

  return changes;
}

const previousProfile = {
  followers: 10240,
  posts: 84,
  bio: 'Helping creators grow faster',
  displayName: 'Creator Ops',
};

const currentProfile = {
  followers: 10980,
  posts: 85,
  bio: 'Helping creators grow faster with better workflows',
  displayName: 'Creator Ops',
};

console.log(diffProfile(previousProfile, currentProfile));

That pattern scales well because each diff type stays simple.

Then you can route the resulting changes into alerts, digests, dashboards, or logs.

Python Version: Same Diff Model, Easy to Batch

In Python I use almost the same logic, just in a more batch-friendly format.

def normalize_text(value):
    return ' '.join((value or '').split())


def diff_number(previous, current, min_absolute=0, min_percent=0):
    prev = float(previous or 0)
    curr = float(current or 0)
    delta = curr - prev
    percent = abs(delta / prev) * 100 if prev > 0 else 0

    changed = abs(delta) >= min_absolute or percent >= min_percent

    return {
        'changed': changed,
        'previous': prev,
        'current': curr,
        'delta': delta,
        'percent': round(percent, 2),
    }


def diff_text(previous, current):
    prev = normalize_text(previous)
    curr = normalize_text(current)
    return {
        'changed': prev != curr,
        'previous': prev,
        'current': curr,
    }


def diff_set(previous=None, current=None):
    prev = set(previous or [])
    curr = set(current or [])

    added = sorted(curr - prev)
    removed = sorted(prev - curr)

    return {
        'changed': bool(added or removed),
        'added': added,
        'removed': removed,
    }


def diff_profile(previous, current):
    followers = diff_number(previous.get('followers'), current.get('followers'), min_absolute=100, min_percent=5)
    posts = diff_number(previous.get('posts'), current.get('posts'), min_absolute=1)
    bio = diff_text(previous.get('bio'), current.get('bio'))
    display_name = diff_text(previous.get('displayName'), current.get('displayName'))

    changes = []

    if followers['changed']:
        changes.append({'type': 'follower_change', 'severity': 'medium', **followers})

    if posts['changed'] and posts['delta'] > 0:
        changes.append({'type': 'new_post_count', 'severity': 'high', **posts})

    if bio['changed']:
        changes.append({'type': 'bio_updated', 'severity': 'low', **bio})

    if display_name['changed']:
        changes.append({'type': 'display_name_updated', 'severity': 'low', **display_name})

    return changes


previous_profile = {
    'followers': 10240,
    'posts': 84,
    'bio': 'Helping creators grow faster',
    'displayName': 'Creator Ops',
}

current_profile = {
    'followers': 10980,
    'posts': 85,
    'bio': 'Helping creators grow faster with better workflows',
    'displayName': 'Creator Ops',
}

print(diff_profile(previous_profile, current_profile))

The Most Important Thing: Severity Classification

This is where the diff engine becomes a monitoring system instead of just a comparison utility.

I try to classify changes like this:

low: bio text, display name formatting, minor stat movement
medium: meaningful follower movement, comment spikes, important profile metadata changes
high: new post detected, post removed, active campaign shift, landing page change

Once you do that, it becomes much easier to decide:

what goes to Slack immediately
what gets saved for a daily digest
what only belongs in the audit log

Without severity, every change competes for attention equally. That is how monitoring systems become unreadable.

Honest Alternatives

There are a few ways to approach this.

Raw object diff libraries

Great for debugging.

Usually too noisy for social monitoring.

Event sourcing everything

Powerful if you have a larger architecture and a reason to keep full history.

Overkill for many small tools.

Handwritten per-entity diff logic

This is still my favorite for social systems.

It is boring, explicit, and much easier to reason about than magic diff output.

Where SociaVault Fits

This is one of those layers where I want the upstream data source to be the boring part.

I use SociaVault for the public social data collection layer, then keep my own diff logic in application code.

That lets me work on the part that actually creates value: deciding what changed and why it matters.

That is the product problem. Collection is just a dependency.

Final Take

The hardest part of social monitoring is not polling.

It is meaning.

Your diff engine decides whether your product becomes a useful signal layer or a machine for generating trivia.

Normalize first. Use thresholds for numbers. Use text normalization for strings. Use set logic for collections. Add severity.

That combination took my monitoring systems from noisy to actually usable.

And if you want to spend more of your time on that diff logic instead of on collection plumbing, SociaVault is a good place to start.

webdev #monitoring #javascript #python #backend

How I'd Build a Landing Page Change Tracker for Competitor Ads

Olamide Olaniyan — Tue, 05 May 2026 19:19:00 +0000

Ad libraries tell you what the ad says.

The landing page tells you what the company is actually trying to make you do.

That difference matters.

I have seen teams obsess over competitor headlines while ignoring the part that changed first: the page behind the ad.

That is where you notice things like:

a new self-serve CTA
a new pricing angle
a comparison page replacing a generic feature page
a trial push replacing a demo flow
a new proof block added to support a fresh campaign

If you are tracking competitors seriously, ad monitoring without landing-page tracking is incomplete.

So this is the workflow I would build: use public ad data to discover the active landing pages, save structured snapshots of those pages, and diff the important parts over time.

Why Ad Pages Are Better Than General Site Crawling

You can crawl a competitor site broadly.

Sometimes that is useful.

But if your real goal is paid-intel and offer monitoring, ad-linked landing pages are much higher signal.

They tell you what the company is actively paying to promote.

That means the workflow is simpler and the insight is usually sharper.

The Three Things I Track First

I start with only a few fields.

page title
primary H1 or hero headline
primary CTA text
canonical URL

That is enough to detect most meaningful competitor landing-page changes.

You do not need a full DOM diff to start getting value.

JavaScript Version: Pull Ad URLs, Then Diff Key Page Elements

This version uses Facebook and LinkedIn ad data to collect landing page URLs, then fetches and compares a few key HTML elements.

const headers = { 'X-API-Key': process.env.SOCIAVAULT_API_KEY };

async function fetchJson(url) {
  const response = await fetch(url, { headers });
  if (!response.ok) {
    throw new Error(`Request failed with ${response.status}`);
  }
  return response.json();
}

async function fetchHtml(url) {
  const response = await fetch(url, {
    headers: {
      'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
    },
  });
  if (!response.ok) {
    throw new Error(`HTML fetch failed with ${response.status}`);
  }
  return response.text();
}

function extractBetween(html, regex) {
  const match = html.match(regex);
  return match ? match[1].replace(/\s+/g, ' ').trim() : null;
}

function parsePage(html, url) {
  return {
    url,
    title: extractBetween(html, /<title>(.*?)<\/title>/is),
    h1: extractBetween(html, /<h1[^>]*>(.*?)<\/h1>/is),
    cta: extractBetween(html, /<a[^>]*>(Start free|Book a demo|Talk to sales|Try free|Get started)<\/a>/is),
  };
}

function normalizeAds(items = []) {
  return (items || []).map(item => item.url || item.landingPageUrl || item.snapshot?.link_url).filter(Boolean);
}

function diffPage(previous, current) {
  const changes = [];

  for (const field of ['title', 'h1', 'cta']) {
    if ((previous?.[field] || null) !== (current?.[field] || null)) {
      changes.push({
        field,
        previous: previous?.[field] || null,
        current: current?.[field] || null,
      });
    }
  }

  return changes;
}

async function collectLandingPages(company) {
  const [facebookJson, linkedinJson] = await Promise.all([
    fetchJson(
      `https://api.sociavault.com/v1/scrape/facebook-ad-library/company-ads?companyName=${encodeURIComponent(company)}&status=ACTIVE&trim=true`
    ),
    fetchJson(
      `https://api.sociavault.com/v1/scrape/linkedin-ad-library/search?company=${encodeURIComponent(company)}`
    ),
  ]);

  const urls = new Set([
    ...normalizeAds(facebookJson.data),
    ...normalizeAds(linkedinJson.data),
  ]);

  const pages = [];
  for (const url of urls) {
    try {
      const html = await fetchHtml(url);
      pages.push(parsePage(html, url));
    } catch (error) {
      console.error(`Failed to fetch ${url}:`, error.message);
    }
  }

  return pages;
}

const previousPages = {
  'https://example.com/demo': {
    title: 'Book a Demo | Example',
    h1: 'See Example in action',
    cta: 'Book a demo',
  },
};

const currentPages = await collectLandingPages('HubSpot');

for (const page of currentPages) {
  const changes = diffPage(previousPages[page.url], page);
  if (changes.length > 0) {
    console.log(`\nChanges detected for ${page.url}`);
    console.log(changes);
  }
}

This is enough to build a daily tracker that tells you when an ad-linked page changes its core message.

And if you want the public ad layer feeding that workflow without managing it manually, SociaVault makes that part much easier.

Python Version: Good for Scheduled Tracking Jobs

Python works very well for this kind of scheduled page-diff script.

import os
import re
import requests


HEADERS = {'X-API-Key': os.environ['SOCIAVAULT_API_KEY']}


def fetch_json(url):
    response = requests.get(url, headers=HEADERS, timeout=30)
    response.raise_for_status()
    return response.json()


def fetch_html(url):
    response = requests.get(
        url,
        headers={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'},
        timeout=30,
    )
    response.raise_for_status()
    return response.text


def extract_between(html, pattern):
    match = re.search(pattern, html, re.IGNORECASE | re.DOTALL)
    return re.sub(r'\s+', ' ', match.group(1)).strip() if match else None


def parse_page(html, url):
    return {
        'url': url,
        'title': extract_between(html, r'<title>(.*?)</title>'),
        'h1': extract_between(html, r'<h1[^>]*>(.*?)</h1>'),
        'cta': extract_between(html, r'<a[^>]*>(Start free|Book a demo|Talk to sales|Try free|Get started)</a>'),
    }


def normalize_ads(items=None):
    items = items or []
    urls = []
    for item in items:
        url = item.get('url') or item.get('landingPageUrl') or item.get('snapshot', {}).get('link_url')
        if url:
            urls.append(url)
    return urls


def diff_page(previous, current):
    changes = []
    for field in ['title', 'h1', 'cta']:
        old_value = (previous or {}).get(field)
        new_value = current.get(field)
        if old_value != new_value:
            changes.append({'field': field, 'previous': old_value, 'current': new_value})
    return changes


def collect_landing_pages(company):
    facebook = fetch_json(
        f'https://api.sociavault.com/v1/scrape/facebook-ad-library/company-ads?companyName={company}&status=ACTIVE&trim=true'
    )
    linkedin = fetch_json(
        f'https://api.sociavault.com/v1/scrape/linkedin-ad-library/search?company={company}'
    )

    urls = set(normalize_ads(facebook.get('data')) + normalize_ads(linkedin.get('data')))
    pages = []

    for url in urls:
        try:
            html = fetch_html(url)
            pages.append(parse_page(html, url))
        except Exception as error:
            print(f'Failed to fetch {url}: {error}')

    return pages


previous_pages = {
    'https://example.com/demo': {
        'title': 'Book a Demo | Example',
        'h1': 'See Example in action',
        'cta': 'Book a demo',
    }
}

current_pages = collect_landing_pages('HubSpot')

for page in current_pages:
    changes = diff_page(previous_pages.get(page['url']), page)
    if changes:
        print(f'\nChanges detected for {page["url"]}')
        print(changes)

The Best Use Cases for This

This kind of tracker is especially useful for:

spotting pricing pivots earlier
seeing when a competitor moves from demo-led to self-serve
catching new proof points or trust elements in the hero section
identifying when a category starts leaning harder into comparison pages

That is much more actionable than just saving screenshots of ads.

Honest Alternatives

There are a few other ways to do this.

General site crawlers

Useful if you care about broad site changes.

Less focused if your real interest is what paid traffic is being sent to.

Visual diff tools

Great when design changes matter.

Often noisier and heavier than necessary if you only care about message shifts.

Manual weekly review

Fine for small lists.

Easy to forget, and very hard to compare consistently over time.

That is why I usually start with simple structured text diffs and only add visual diffing later if needed.

Final Take

If you want to understand what a competitor is actually pushing, look past the ad and track the page.

That is where the real shift often shows up first.

Use public ad data to find the current destination URLs. Save a few meaningful fields. Diff them over time. That alone gives you a much more useful competitor signal than casual ad-library browsing.

And if you want the ad-side data layer without wiring it all manually, SociaVault is a good place to start.

Then keep the rest simple and boring: fetch, extract, diff, report.

That is enough to make a landing-page change tracker genuinely useful.

webdev #monitoring #python #javascript #marketing

What Facebook Video Transcripts Reveal Before Sales Calls Do

Olamide Olaniyan — Mon, 04 May 2026 16:16:00 +0000

Most teams treat competitor videos like brand content.

They watch a reel, nod a little, maybe save the link, then move on.

I think that leaves a lot of value on the table.

Because once you stop watching those videos as content and start treating them as data, a more useful question shows up:

what are they saying repeatedly, and what does that tell us about the market right now?

That is where transcripts become interesting.

Not because transcripts are glamorous, but because they make short-form video searchable, classifiable, and diffable.

And once you can do that, you can spot changes in:

positioning
offers
objections
proof points
CTA strategy

Often before those changes become obvious in sales calls or broader market chatter.

This is the workflow I would build if I wanted Facebook reels and video posts to become a structured competitor-intelligence source instead of random tabs in a browser.

Why Transcripts Matter More Than Views

Views tell you reach.

Transcripts tell you messaging.

If I am monitoring competitors, messaging is usually more useful.

I want to know:

which promises keep showing up
which customer pains they repeat
what proof they lead with
whether the CTA changed this week
whether they are moving upmarket, downmarket, or more self-serve

That is what transcripts make easier to capture.

The Workflow I Like

The workflow is simple.

Collect public Facebook video or reel URLs
Pull the transcript for each one
Split transcript text into short statements
Classify those statements into buckets
Compare those buckets across time

You do not need a giant NLP stack to get useful output.

You just need enough structure to stop relying on memory.

JavaScript Version: Transcript Bucketing for Competitor Research

This starts with the Facebook transcript route and then turns transcript text into reusable messaging buckets.

const headers = {
  'X-API-Key': process.env.SOCIAVAULT_API_KEY,
};

async function fetchJson(url) {
  const response = await fetch(url, { headers });
  if (!response.ok) {
    throw new Error(`Request failed with ${response.status}`);
  }
  return response.json();
}

function extractTranscript(data = {}) {
  if (typeof data === 'string') return data;
  if (typeof data.transcript === 'string') return data.transcript;
  if (typeof data.text === 'string') return data.text;
  if (Array.isArray(data.captions)) {
    return data.captions.map(item => item.text || item.caption || '').join(' ');
  }
  return '';
}

function splitStatements(text) {
  return text
    .split(/[.!?\n]+/)
    .map(part => part.trim())
    .filter(Boolean)
    .filter(part => part.length > 20);
}

function classifyStatement(statement) {
  const value = statement.toLowerCase();

  if (/(struggling|frustrated|hard|difficult|problem|stuck|wasting time)/.test(value)) return 'pain';
  if (/(customers|teams|creators|brands|founders|agencies)/.test(value)) return 'audience';
  if (/(results|case study|revenue|saved|grew|increase|improved)/.test(value)) return 'proof';
  if (/(free trial|book a demo|get started|sign up|dm us|click the link)/.test(value)) return 'cta';
  if (/(price|pricing|plan|subscription|monthly|annual|cost)/.test(value)) return 'offer';
  return 'positioning';
}

async function analyzeTranscript(url) {
  const json = await fetchJson(
    `https://api.sociavault.com/v1/scrape/facebook/post/transcript?url=${encodeURIComponent(url)}`
  );

  const transcript = extractTranscript(json.data || {});
  const statements = splitStatements(transcript);
  const buckets = {
    pain: [],
    audience: [],
    proof: [],
    offer: [],
    cta: [],
    positioning: [],
  };

  for (const statement of statements) {
    buckets[classifyStatement(statement)].push(statement);
  }

  return {
    url,
    transcriptLength: transcript.length,
    bucketCounts: Object.fromEntries(Object.entries(buckets).map(([key, value]) => [key, value.length])),
    examples: Object.fromEntries(Object.entries(buckets).map(([key, value]) => [key, value.slice(0, 3)])),
  };
}

analyzeTranscript('https://www.facebook.com/reel/1535656380759655')
  .then(report => console.log(report))
  .catch(error => console.error(error));

This is already enough to make transcript review much more systematic.

Instead of saying "their content feels more salesy lately," I can actually show which buckets increased.

Python Version: Compare Multiple Videos Over Time

Once you want weekly or monthly tracking, Python is a comfortable way to batch it.

import os
import re
import requests
from collections import Counter


HEADERS = {'X-API-Key': os.environ['SOCIAVAULT_API_KEY']}


def fetch_json(url):
    response = requests.get(url, headers=HEADERS, timeout=30)
    response.raise_for_status()
    return response.json()


def extract_transcript(data):
    if isinstance(data, str):
        return data
    if isinstance(data, dict):
        if isinstance(data.get('transcript'), str):
            return data['transcript']
        if isinstance(data.get('text'), str):
            return data['text']
        if isinstance(data.get('captions'), list):
            return ' '.join((item.get('text') or item.get('caption') or '') for item in data['captions'])
    return ''


def split_statements(text):
    parts = re.split(r'[.!?\n]+', text)
    return [part.strip() for part in parts if part.strip() and len(part.strip()) > 20]


def classify_statement(statement):
    value = statement.lower()
    if re.search(r'(struggling|frustrated|hard|difficult|problem|stuck|wasting time)', value):
        return 'pain'
    if re.search(r'(customers|teams|creators|brands|founders|agencies)', value):
        return 'audience'
    if re.search(r'(results|case study|revenue|saved|grew|increase|improved)', value):
        return 'proof'
    if re.search(r'(free trial|book a demo|get started|dm us|click the link|sign up)', value):
        return 'cta'
    if re.search(r'(price|pricing|plan|subscription|monthly|annual|cost)', value):
        return 'offer'
    return 'positioning'


def analyze_transcript(url):
    json_data = fetch_json(
        f'https://api.sociavault.com/v1/scrape/facebook/post/transcript?url={url}'
    )
    transcript = extract_transcript(json_data.get('data') or {})
    statements = split_statements(transcript)

    buckets = Counter(classify_statement(statement) for statement in statements)
    examples = {}
    for bucket in ['pain', 'audience', 'proof', 'offer', 'cta', 'positioning']:
        examples[bucket] = [statement for statement in statements if classify_statement(statement) == bucket][:3]

    return {
        'url': url,
        'transcriptLength': len(transcript),
        'bucketCounts': dict(buckets),
        'examples': examples,
    }


urls = [
    'https://www.facebook.com/reel/1535656380759655',
    'https://www.facebook.com/reel/1535656380759656',
]

reports = [analyze_transcript(url) for url in urls]
print(reports)

That is enough to answer questions like:

did their CTA mix change?
are they leaning harder into proof recently?
are they talking about pricing more often?
did they shift from pain-led messaging to offer-led messaging?

That is useful market intelligence from public video content.

Where This Gets Especially Useful

I like transcript analysis most in four cases.

1. Competitor sales messaging

Watch the words they keep repeating.

2. Offer changes

Notice when more pricing, trial, or package language starts showing up.

3. Objection mining

Competitor videos often answer customer concerns directly.

4. Content planning

You can see which positioning angles the market is saturating.

That helps your own team avoid parroting everyone else.

Honest Alternatives

You can absolutely do this manually.

For a handful of videos, manual review is fine.

But once you are tracking multiple competitors across time, transcripts are what make the workflow scalable.

You can also push this further with embeddings or LLM summarization later. I would just avoid starting there.

The basic transcript bucketing layer is already useful and much easier to trust.

Final Take

Competitor videos are not just content samples. They are messaging assets hiding in plain sight.

Transcripts turn them into something you can actually analyze over time.

If you want the public transcript layer without building the collection plumbing from scratch, SociaVault is a good place to start.

Then do the part that actually matters: classify what they keep saying, compare it across time, and use that signal before the market feels obvious.

That is usually where the edge is.

webdev #python #javascript #marketing #competitiveanalysis

The Hard Part of Social Monitoring Isn't Collection. It's Deciding What Changed

Olamide Olaniyan — Sun, 03 May 2026 19:15:00 +0000

The first version of my social monitoring stack collected a lot of data.

That was not the same thing as producing insight.

Every polling cycle gave me fresh JSON. New stats. New posts. New comments. Profile snapshots. Search results.

It felt productive.

It was mostly noise.

Because the real difficulty in social monitoring is not getting the data.

It is answering the question:

what changed here that is actually worth somebody's attention?

That is the whole product problem.

This post is about the change-detection layer I wish I had built earlier: how I classify changes, which ones deserve alerts, how I implement the logic in JavaScript and Python, and how a system like SociaVault helps by turning collection into the boring part.

Collection Is Table Stakes

By now, most people reading Dev.to know how to hit an API, queue jobs, or scrape a page.

That part is solvable.

What usually breaks in monitoring products is everything after collection:

too many low-signal alerts
missing the genuinely important changes
no distinction between fresh data and meaningful change
users muting the channel because it became wallpaper

That is why I think of monitoring as a ranking problem, not just a fetching problem.

The Four Questions I Ask About Every Change

When a new state arrives, I ask four things.

1. Is it real?

Could be an API inconsistency, formatting difference, temporary missing field, or unstable count.

2. Is it meaningful?

A 0.2% follower change is usually not meaningful.

3. Is it urgent?

A new campaign launch might need an alert today. A bio edit usually does not.

4. Who cares?

Growth, sales, creator ops, PR, and product teams care about different things.

Once I started filtering through those four questions, the quality of the monitoring output improved immediately.

The Categories I Use Most Often

For most social monitoring workflows, I bucket changes into five groups.

Structural changes

New post, deleted post, new comment thread, new ad, profile field changed.

Metric changes

Follower movement, engagement spikes, view jumps, comment growth.

Message changes

New CTA, new headline, pricing language, new offer positioning.

Destination changes

New landing page, changed route, changed CTA destination.

Trend changes

Something that is not urgent once, but becomes meaningful over several snapshots.

This matters because each group deserves a different alert policy.

JavaScript Version: A Scoring Layer for Change Detection

I like to turn raw change records into scored events.

That keeps the rest of the pipeline simple.

function scoreChange(change) {
  let score = 0;

  if (change.type === 'new_post') score += 70;
  if (change.type === 'new_ad') score += 80;
  if (change.type === 'landing_page_changed') score += 85;
  if (change.type === 'cta_changed') score += 75;
  if (change.type === 'follower_change') score += 40;
  if (change.type === 'bio_updated') score += 20;

  if (change.percentChange && change.percentChange >= 10) score += 15;
  if (change.absoluteDelta && Math.abs(change.absoluteDelta) >= 1000) score += 15;
  if (change.platform === 'linkedin') score += 5;

  let priority = 'log';
  if (score >= 80) priority = 'alert';
  else if (score >= 50) priority = 'digest';

  return { ...change, score, priority };
}

function detectMeaningfulChanges(previous, current) {
  const changes = [];

  if ((current.posts || 0) > (previous.posts || 0)) {
    changes.push({
      type: 'new_post',
      platform: current.platform,
      absoluteDelta: current.posts - previous.posts,
    });
  }

  const prevFollowers = previous.followers || 0;
  const currFollowers = current.followers || 0;
  const followerDelta = currFollowers - prevFollowers;
  const percentChange = prevFollowers > 0
    ? Math.abs(followerDelta / prevFollowers) * 100
    : 0;

  if (Math.abs(followerDelta) >= 500 || percentChange >= 5) {
    changes.push({
      type: 'follower_change',
      platform: current.platform,
      absoluteDelta: followerDelta,
      percentChange: Number(percentChange.toFixed(2)),
    });
  }

  if ((previous.bio || '').trim() !== (current.bio || '').trim()) {
    changes.push({
      type: 'bio_updated',
      platform: current.platform,
      oldValue: previous.bio,
      newValue: current.bio,
    });
  }

  return changes.map(scoreChange);
}

const previous = {
  platform: 'tiktok',
  followers: 20200,
  posts: 42,
  bio: 'Helping brands find creators',
};

const current = {
  platform: 'tiktok',
  followers: 21480,
  posts: 43,
  bio: 'Helping brands find creators faster',
};

console.log(detectMeaningfulChanges(previous, current));

Now the rest of the system can behave differently based on priority:

alert goes to Slack now
digest goes into the daily summary
log stays in the audit trail

That separation is what makes the system survivable.

Python Version: Same Model, Better for Batch Jobs

If you are doing scheduled monitoring in Python, the exact same approach works.

def score_change(change):
    score = 0

    if change['type'] == 'new_post':
        score += 70
    if change['type'] == 'new_ad':
        score += 80
    if change['type'] == 'landing_page_changed':
        score += 85
    if change['type'] == 'cta_changed':
        score += 75
    if change['type'] == 'follower_change':
        score += 40
    if change['type'] == 'bio_updated':
        score += 20

    if change.get('percentChange', 0) >= 10:
        score += 15
    if abs(change.get('absoluteDelta', 0)) >= 1000:
        score += 15
    if change.get('platform') == 'linkedin':
        score += 5

    priority = 'log'
    if score >= 80:
        priority = 'alert'
    elif score >= 50:
        priority = 'digest'

    return {**change, 'score': score, 'priority': priority}


def detect_meaningful_changes(previous, current):
    changes = []

    if current.get('posts', 0) > previous.get('posts', 0):
        changes.append({
            'type': 'new_post',
            'platform': current.get('platform'),
            'absoluteDelta': current['posts'] - previous['posts'],
        })

    prev_followers = previous.get('followers', 0)
    curr_followers = current.get('followers', 0)
    follower_delta = curr_followers - prev_followers
    percent_change = abs(follower_delta / prev_followers) * 100 if prev_followers > 0 else 0

    if abs(follower_delta) >= 500 or percent_change >= 5:
        changes.append({
            'type': 'follower_change',
            'platform': current.get('platform'),
            'absoluteDelta': follower_delta,
            'percentChange': round(percent_change, 2),
        })

    if (previous.get('bio') or '').strip() != (current.get('bio') or '').strip():
        changes.append({
            'type': 'bio_updated',
            'platform': current.get('platform'),
            'oldValue': previous.get('bio'),
            'newValue': current.get('bio'),
        })

    return [score_change(change) for change in changes]


previous = {
    'platform': 'tiktok',
    'followers': 20200,
    'posts': 42,
    'bio': 'Helping brands find creators',
}

current = {
    'platform': 'tiktok',
    'followers': 21480,
    'posts': 43,
    'bio': 'Helping brands find creators faster',
}

print(detect_meaningful_changes(previous, current))

The Best Monitoring Systems Have Memory

This is the part that gets overlooked a lot.

A meaningful change is often not something visible in one snapshot.

Sometimes it only becomes meaningful when you compare several snapshots over time.

Examples:

the same CTA appears across more and more ads
landing pages consolidate around one offer
comment velocity slows over several intervals
follower growth looks stalled for weeks, not hours

That is why I treat change detection and trend detection as related but separate layers.

One looks for immediate events.

The other looks for patterns.

Honest Alternatives

There are a few ways to handle this depending on the team.

Alert on everything

Fastest to build.

Usually unbearable after a week.

Manual review dashboards

Good for analysts.

Bad for busy teams that need push delivery.

Rule-based scoring plus digests

This is still my favorite starting point.

It is simple, explainable, and easy to tune.

If you want something smarter later, add ML or embeddings after the rule-based system proves useful.

Where SociaVault Fits

This is exactly the kind of problem where I want SociaVault underneath the system, not inside the part I am constantly editing.

I want the public social data collection layer to be stable.

That way I can spend time on the hard product question: deciding what changed enough to deserve attention.

That is the layer users actually pay attention to.

Final Take

Collection gets you data.

Change detection gets you value.

If your monitoring system feels noisy, the answer is usually not better scraping. It is better judgment in the layer that decides what matters.

Score the changes. Route by urgency. Separate events from trends.

That is what finally made my monitoring systems feel useful instead of just active.

And if you want the collection layer to be the boring part so you can focus on that judgment layer, SociaVault is a good place to start.

webdev #monitoring #backend #python #javascript

Building a Daily Competitor Digest Instead of Yet Another Dashboard

Olamide Olaniyan — Sat, 02 May 2026 19:14:00 +0000

I like dashboards.

I just do not trust myself to check them consistently.

That was the whole reason I built a daily competitor digest.

I had all the data: competitor ads, landing pages, CTA shifts, campaign launches. But the insight was trapped inside tools that only became useful if someone remembered to open them.

That is the flaw with a lot of competitive intelligence stacks.

They optimize for exploration, not attention.

If you want a team to notice important competitor changes, the system should show up where they already work. For most teams, that means Slack or email. Not another tab.

So instead of building yet another competitor monitoring dashboard, I built a daily digest that answers one question every morning:

what changed yesterday that is actually worth knowing?

This post walks through the architecture, the diff logic, the data sources, and the JavaScript and Python code I would use to build it.

What Belongs in a Competitor Digest

The digest should not be a dump of every signal you can collect.

It should be a shortlist of changes that matter.

The categories I care about most are:

new ads launched
new landing pages detected
meaningful CTA changes
platform expansion or reactivation
unusually large jumps in active creative count

That is enough to tell a growth team, founder, or product marketer where to look next.

Why a Digest Beats a Dashboard for Busy Teams

Dashboards are good for exploration.

Digests are good for follow-through.

A dashboard says, "go look around."

A digest says, "here are the three things that changed, and here is why they matter."

That difference is huge in practice.

Most teams do not have a competitor data problem. They have a prioritization problem.

The Architecture I Like Best

Keep it boring.

That is a compliment.

Scheduler
  -> Collect today's snapshots
  -> Compare with yesterday's snapshots
  -> Build a short summary
  -> Send to Slack or email

That is it.

You do not need an overbuilt analytics system to get value from this.

JavaScript Version: Daily Digest From Ad Library Snapshots

This version uses public ad data from Facebook, Google, LinkedIn, and Reddit, then compares today's snapshot with yesterday's stored JSON.

import fs from 'fs/promises';

const headers = { 'X-API-Key': process.env.SOCIAVAULT_API_KEY };
const SLACK_WEBHOOK = process.env.SLACK_WEBHOOK_URL;

const competitors = [
  { name: 'HubSpot', domain: 'hubspot.com', redditQuery: 'HubSpot' },
  { name: 'Notion', domain: 'notion.so', redditQuery: 'Notion' },
];

async function fetchJson(url) {
  const response = await fetch(url, { headers });
  if (!response.ok) {
    throw new Error(`Request failed with ${response.status}: ${url}`);
  }
  return response.json();
}

function normalizeAds(items = []) {
  return (items || []).map(item => ({
    headline: item.headline || item.title || item.snapshot?.title || '',
    body: item.body || item.text || item.snapshot?.body?.markup || '',
    cta: item.cta || item.call_to_action || item.snapshot?.cta_text || '',
    url: item.url || item.landingPageUrl || item.snapshot?.link_url || '',
  }));
}

async function collectSnapshot(competitor) {
  const [facebook, google, linkedin, reddit] = await Promise.all([
    fetchJson(
      `https://api.sociavault.com/v1/scrape/facebook-ad-library/company-ads?companyName=${encodeURIComponent(competitor.name)}&status=ACTIVE&trim=true`
    ),
    fetchJson(
      `https://api.sociavault.com/v1/scrape/google-ad-library/company-ads?domain=${encodeURIComponent(competitor.domain)}&region=US`
    ),
    fetchJson(
      `https://api.sociavault.com/v1/scrape/linkedin-ad-library/search?company=${encodeURIComponent(competitor.name)}`
    ),
    fetchJson(
      `https://api.sociavault.com/v1/scrape/reddit/ads/search?query=${encodeURIComponent(competitor.redditQuery)}`
    ),
  ]);

  return {
    capturedAt: new Date().toISOString(),
    competitor: competitor.name,
    facebook: normalizeAds(facebook.data),
    google: normalizeAds(google.data),
    linkedin: normalizeAds(linkedin.data),
    reddit: normalizeAds(reddit.data),
  };
}

function uniqueValues(items, field) {
  return new Set(items.map(item => item[field]).filter(Boolean));
}

function diffSnapshots(today, yesterday) {
  const lines = [];

  for (const platform of ['facebook', 'google', 'linkedin', 'reddit']) {
    const current = today[platform] || [];
    const previous = yesterday?.[platform] || [];

    const currentUrls = uniqueValues(current, 'url');
    const previousUrls = uniqueValues(previous, 'url');
    const currentCtas = uniqueValues(current, 'cta');
    const previousCtas = uniqueValues(previous, 'cta');

    const newLandingPages = [...currentUrls].filter(url => !previousUrls.has(url));
    const newCtas = [...currentCtas].filter(cta => !previousCtas.has(cta));
    const countDelta = current.length - previous.length;

    if (countDelta > 0) {
      lines.push(`${platform}: ${countDelta} new active ads`);
    }

    if (newLandingPages.length > 0) {
      lines.push(`${platform}: ${newLandingPages.length} new landing page${newLandingPages.length > 1 ? 's' : ''}`);
    }

    if (newCtas.length > 0) {
      lines.push(`${platform}: new CTA signals -> ${newCtas.slice(0, 3).join(', ')}`);
    }

    if (previous.length === 0 && current.length > 0) {
      lines.push(`${platform}: platform reactivated or newly detected`);
    }
  }

  return lines;
}

async function saveSnapshot(filePath, snapshot) {
  await fs.writeFile(filePath, JSON.stringify(snapshot, null, 2));
}

async function loadSnapshot(filePath) {
  try {
    const raw = await fs.readFile(filePath, 'utf8');
    return JSON.parse(raw);
  } catch {
    return null;
  }
}

async function sendSlackDigest(text) {
  await fetch(SLACK_WEBHOOK, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ text }),
  });
}

for (const competitor of competitors) {
  const filePath = `./snapshots/${competitor.name.toLowerCase().replace(/\s+/g, '-')}.json`;
  const yesterday = await loadSnapshot(filePath);
  const today = await collectSnapshot(competitor);
  const lines = diffSnapshots(today, yesterday);

  const digest = lines.length
    ? `Daily competitor digest: ${competitor.name}\n\n- ${lines.join('\n- ')}`
    : `Daily competitor digest: ${competitor.name}\n\n- No meaningful public ad changes detected.`;

  console.log(digest);
  await sendSlackDigest(digest);
  await saveSnapshot(filePath, today);
}

This is intentionally simple.

You can make it smarter later with scoring, AI summaries, ownership routing, or landing-page HTML diffs.

But this version is already useful.

Python Version: Same Digest, Same Core Logic

If you prefer Python for automation scripts and scheduled jobs, the same design works well there too.

import json
import os
from pathlib import Path
from datetime import datetime, timezone

import requests


HEADERS = {'X-API-Key': os.environ['SOCIAVAULT_API_KEY']}
SLACK_WEBHOOK = os.environ['SLACK_WEBHOOK_URL']
SNAPSHOT_DIR = Path('./snapshots')
SNAPSHOT_DIR.mkdir(exist_ok=True)

COMPETITORS = [
    {'name': 'HubSpot', 'domain': 'hubspot.com', 'reddit_query': 'HubSpot'},
    {'name': 'Notion', 'domain': 'notion.so', 'reddit_query': 'Notion'},
]


def fetch_json(url):
    response = requests.get(url, headers=HEADERS, timeout=30)
    response.raise_for_status()
    return response.json()


def normalize_ads(items=None):
    items = items or []
    normalized = []
    for item in items:
        normalized.append({
            'headline': item.get('headline') or item.get('title') or item.get('snapshot', {}).get('title', ''),
            'body': item.get('body') or item.get('text') or item.get('snapshot', {}).get('body', {}).get('markup', ''),
            'cta': item.get('cta') or item.get('call_to_action') or item.get('snapshot', {}).get('cta_text', ''),
            'url': item.get('url') or item.get('landingPageUrl') or item.get('snapshot', {}).get('link_url', ''),
        })
    return normalized


def collect_snapshot(competitor):
    facebook = fetch_json(
        f"https://api.sociavault.com/v1/scrape/facebook-ad-library/company-ads?companyName={competitor['name']}&status=ACTIVE&trim=true"
    )
    google = fetch_json(
        f"https://api.sociavault.com/v1/scrape/google-ad-library/company-ads?domain={competitor['domain']}&region=US"
    )
    linkedin = fetch_json(
        f"https://api.sociavault.com/v1/scrape/linkedin-ad-library/search?company={competitor['name']}"
    )
    reddit = fetch_json(
        f"https://api.sociavault.com/v1/scrape/reddit/ads/search?query={competitor['reddit_query']}"
    )

    return {
        'capturedAt': datetime.now(timezone.utc).isoformat(),
        'competitor': competitor['name'],
        'facebook': normalize_ads(facebook.get('data')),
        'google': normalize_ads(google.get('data')),
        'linkedin': normalize_ads(linkedin.get('data')),
        'reddit': normalize_ads(reddit.get('data')),
    }


def diff_snapshots(today, yesterday):
    lines = []

    for platform in ['facebook', 'google', 'linkedin', 'reddit']:
        current = today.get(platform, [])
        previous = yesterday.get(platform, []) if yesterday else []

        current_urls = {item['url'] for item in current if item.get('url')}
        previous_urls = {item['url'] for item in previous if item.get('url')}
        current_ctas = {item['cta'] for item in current if item.get('cta')}
        previous_ctas = {item['cta'] for item in previous if item.get('cta')}

        new_urls = current_urls - previous_urls
        new_ctas = current_ctas - previous_ctas
        count_delta = len(current) - len(previous)

        if count_delta > 0:
            lines.append(f'{platform}: {count_delta} new active ads')
        if new_urls:
            lines.append(f'{platform}: {len(new_urls)} new landing pages')
        if new_ctas:
            lines.append(f"{platform}: new CTA signals -> {', '.join(list(new_ctas)[:3])}")
        if not previous and current:
            lines.append(f'{platform}: platform reactivated or newly detected')

    return lines


def load_snapshot(path):
    if not path.exists():
        return None
    return json.loads(path.read_text())


def save_snapshot(path, snapshot):
    path.write_text(json.dumps(snapshot, indent=2))


def send_slack_digest(text):
    requests.post(SLACK_WEBHOOK, json={'text': text}, timeout=30).raise_for_status()


for competitor in COMPETITORS:
    path = SNAPSHOT_DIR / f"{competitor['name'].lower().replace(' ', '-')}.json"
    yesterday = load_snapshot(path)
    today = collect_snapshot(competitor)
    lines = diff_snapshots(today, yesterday)

    digest = (
        f"Daily competitor digest: {competitor['name']}\n\n- " + '\n- '.join(lines)
        if lines else
        f"Daily competitor digest: {competitor['name']}\n\n- No meaningful public ad changes detected."
    )

    print(digest)
    send_slack_digest(digest)
    save_snapshot(path, today)

What I Would Add Next

Once the basic digest works, the next useful upgrades are usually:

HTML diffs for landing pages
scoring so only the highest-signal changes get included
AI summarization for longer daily reports
separate digests for paid media and organic content
weekly rollups for leadership

But do not start there.

Get the daily change detection working first.

That is where the value lives.

Honest Alternatives

There are cases where a digest is not the best default.

Real-time alerts

Better for crisis monitoring, live launch tracking, or brand safety.

Dashboards

Better for analysts who want to explore everything.

Weekly reports

Better for leadership teams that do not want daily noise.

For most growth and competitive-intel teams, though, the daily digest is the sweet spot.

It keeps the system useful without forcing everyone to become a dashboard tourist.

Where SociaVault Fits

This workflow gets a lot easier when you are not manually checking four different public ad libraries yourself.

That is why I like using SociaVault as the public data layer here. It lets me spend my time on the snapshot, diff, and digest logic instead of the collection layer.

That is the split I want as an engineer.

Collection is table stakes. Delivery is where the product becomes useful.

Final Take

The best competitor monitoring system is not the one with the fanciest dashboard.

It is the one your team actually reads.

That is why I keep coming back to a daily digest. It turns competitor data into a habit instead of a homework assignment.

If you want to build one without wiring up every ad library manually, SociaVault is a solid place to start.

Then keep the rest boring: collect, diff, summarize, deliver.

That is enough to make competitor monitoring much more useful than most teams expect.

webdev #python #javascript #monitoring #marketing

I Compared Facebook, Google, LinkedIn, and Reddit Ads to See Where Messaging Changes First

Olamide Olaniyan — Fri, 01 May 2026 19:13:00 +0000

I used to look at ad libraries the lazy way.

Open one platform. Skim a few ads. Say something vague like "they seem more aggressive lately." Close the tab. Move on.

That is not analysis. That is professional browsing.

What I actually wanted to know was much more specific:

when a company changes its message, which platform shows the shift first?

That turns out to be a useful question.

Because messaging changes often show up publicly before the company has fully rolled them out everywhere. A new CTA appears on LinkedIn. A stronger price-led angle shows up on Facebook. Reddit ads start testing sharper pain-point language. Google advertiser results start reflecting a new intent cluster.

If you track those changes consistently, you can often spot a repositioning move earlier than teams who only look at one platform at a time.

So this post is the workflow I use to compare ad messaging shifts across Facebook, Google, LinkedIn, and Reddit, plus JavaScript and Python examples you can actually adapt.

Why This Is More Useful Than "Ad Spy"

Most ad-spy workflows are really screenshot collections.

That is fine if all you want is creative inspiration.

But if you care about strategy, you should be asking better questions:

Which platform gets the new message first?
Which platform gets the strongest commercial CTA?
Where does the company test narrative before broad rollout?
Which angle stays niche-specific versus becoming company-wide?

Those are not creative questions.

They are go-to-market questions.

The Pattern I Keep Seeing

This is not a law, but it is a pattern I have seen often enough to trust it as a starting point:

LinkedIn tends to reveal deliberate B2B positioning changes early
Facebook tends to show broader offer and creative scaling
Google reflects higher-intent buyer language once the company is ready to capture demand
Reddit often exposes sharper, less polished pain-point phrasing

That means the first place a message changes is not always the biggest platform.

Sometimes it is the platform best suited to the earliest test.

That is what makes cross-platform comparison worth doing.

The Four Message Buckets I Start With

You can get more sophisticated later. I usually start simple.

My first-pass buckets are:

price-led: words like pricing, save, cost, budget, annual
demo-led: book a demo, talk to sales, see it in action
proof-led: case study, trusted by, results, ROI
pain-led: manual, slow, messy, wasted time, switch

That is enough to spot a meaningful shift in tone and intent.

JavaScript Version: Compare Messaging Across Ad Libraries

This version pulls public ad data from four platforms, normalizes the text, and counts message buckets.

const headers = { 'X-API-Key': process.env.SOCIAVAULT_API_KEY };

async function fetchJson(url) {
  const response = await fetch(url, { headers });
  if (!response.ok) {
    throw new Error(`Request failed with ${response.status}`);
  }
  return response.json();
}

function normalizeAds(items = []) {
  return (items || []).map(item => ({
    headline: item.headline || item.title || item.snapshot?.title || '',
    body: item.body || item.text || item.snapshot?.body?.markup || '',
    cta: item.cta || item.call_to_action || item.snapshot?.cta_text || '',
  }));
}

function classifyMessage(ad) {
  const text = `${ad.headline} ${ad.body} ${ad.cta}`.toLowerCase();

  if (/(pricing|save|cost|budget|annual)/.test(text)) return 'price-led';
  if (/(book a demo|talk to sales|see it in action|request demo)/.test(text)) return 'demo-led';
  if (/(case study|trusted by|roi|results|used by)/.test(text)) return 'proof-led';
  if (/(manual|slow|messy|wasted time|switch)/.test(text)) return 'pain-led';
  return 'other';
}

function summarizeMessages(ads) {
  const summary = {};

  for (const ad of ads) {
    const bucket = classifyMessage(ad);
    summary[bucket] = (summary[bucket] || 0) + 1;
  }

  return summary;
}

async function compareMessaging(company, domain) {
  const [facebookJson, googleJson, linkedinJson, redditJson] = await Promise.all([
    fetchJson(
      `https://api.sociavault.com/v1/scrape/facebook-ad-library/company-ads?companyName=${encodeURIComponent(company)}&status=ACTIVE&trim=true`
    ),
    fetchJson(
      `https://api.sociavault.com/v1/scrape/google-ad-library/company-ads?domain=${encodeURIComponent(domain)}&region=US`
    ),
    fetchJson(
      `https://api.sociavault.com/v1/scrape/linkedin-ad-library/search?company=${encodeURIComponent(company)}`
    ),
    fetchJson(
      `https://api.sociavault.com/v1/scrape/reddit/ads/search?query=${encodeURIComponent(company)}`
    ),
  ]);

  const platformData = {
    facebook: normalizeAds(facebookJson.data),
    google: normalizeAds(googleJson.data),
    linkedin: normalizeAds(linkedinJson.data),
    reddit: normalizeAds(redditJson.data),
  };

  for (const [platform, ads] of Object.entries(platformData)) {
    console.log(`\n${platform.toUpperCase()}`);
    console.log(summarizeMessages(ads));
  }
}

compareMessaging('HubSpot', 'hubspot.com').catch(console.error);

This will not give you perfect strategic truth.

It will give you something very useful: a structured way to compare how message categories are distributed by platform right now.

If you run it repeatedly and save snapshots, it gets even better.

And if you want a simpler way to get the public ad data feeding that analysis, SociaVault is a very practical layer for it.

Python Version: Better for Batch Analysis

If you want to run this as a scheduled analyst job or notebook-friendly script, Python is a good fit.

import os
import requests


HEADERS = {'X-API-Key': os.environ['SOCIAVAULT_API_KEY']}


def fetch_json(url):
    response = requests.get(url, headers=HEADERS, timeout=30)
    response.raise_for_status()
    return response.json()


def normalize_ads(items=None):
    items = items or []
    normalized = []
    for item in items:
        normalized.append({
            'headline': item.get('headline') or item.get('title') or item.get('snapshot', {}).get('title', ''),
            'body': item.get('body') or item.get('text') or item.get('snapshot', {}).get('body', {}).get('markup', ''),
            'cta': item.get('cta') or item.get('call_to_action') or item.get('snapshot', {}).get('cta_text', ''),
        })
    return normalized


def classify_message(ad):
    text = f"{ad['headline']} {ad['body']} {ad['cta']}".lower()

    if any(term in text for term in ['pricing', 'save', 'cost', 'budget', 'annual']):
        return 'price-led'
    if any(term in text for term in ['book a demo', 'talk to sales', 'see it in action', 'request demo']):
        return 'demo-led'
    if any(term in text for term in ['case study', 'trusted by', 'roi', 'results', 'used by']):
        return 'proof-led'
    if any(term in text for term in ['manual', 'slow', 'messy', 'wasted time', 'switch']):
        return 'pain-led'
    return 'other'


def summarize_messages(ads):
    summary = {}
    for ad in ads:
        bucket = classify_message(ad)
        summary[bucket] = summary.get(bucket, 0) + 1
    return summary


def compare_messaging(company, domain):
    facebook = fetch_json(
        f'https://api.sociavault.com/v1/scrape/facebook-ad-library/company-ads?companyName={company}&status=ACTIVE&trim=true'
    )
    google = fetch_json(
        f'https://api.sociavault.com/v1/scrape/google-ad-library/company-ads?domain={domain}&region=US'
    )
    linkedin = fetch_json(
        f'https://api.sociavault.com/v1/scrape/linkedin-ad-library/search?company={company}'
    )
    reddit = fetch_json(
        f'https://api.sociavault.com/v1/scrape/reddit/ads/search?query={company}'
    )

    platform_data = {
        'facebook': normalize_ads(facebook.get('data')),
        'google': normalize_ads(google.get('data')),
        'linkedin': normalize_ads(linkedin.get('data')),
        'reddit': normalize_ads(reddit.get('data')),
    }

    for platform, ads in platform_data.items():
        print(f'\n{platform.upper()}')
        print(summarize_messages(ads))


compare_messaging('HubSpot', 'hubspot.com')

What I Look For in the Output

This is the important part.

Do not stop at counting buckets.

I usually ask:

Is one message type concentrated on a single platform?
Did a new message category appear on LinkedIn before Facebook?
Is Google reflecting the same buyer language as the social platforms?
Does Reddit show rougher, more direct pain language than the rest?

That is where the strategic insight starts.

A Practical Example

Say a competitor used to run mostly demo-led language everywhere.

Then you notice:

LinkedIn shifts toward proof-led ads
Facebook starts mixing in price-led copy
Reddit ads get more pain-led
Google advertiser results show new landing pages around self-serve onboarding

That is probably not random.

It often means the company is repositioning or testing a new go-to-market motion.

That is exactly the kind of thing I want to know before it becomes obvious in the market.

Honest Alternatives

There are a few other ways to do this.

Manual ad-library review

Fine for one competitor once in a while.

Weak if you care about repeated comparison and timing.

Full NLP or embeddings on ad copy

Very powerful.

Usually more complexity than you need at the start.

Single-platform monitoring

Useful, but incomplete.

I think this question becomes much more interesting only when the platforms are compared together.

Final Take

The interesting question is not just what ads a company is running.

It is where their message changes first.

That is the signal I care about because it often tells you where strategy is moving before the rollout is complete.

If you want to build that kind of cross-platform comparison workflow without stitching together four different public ad systems by hand, SociaVault is a very good place to start.

Then keep the rest simple: normalize, bucket, compare, repeat.

That alone can tell you a surprising amount about where a competitor is heading.

webdev #marketing #python #javascript #analytics

Building a Creator Due Diligence Workflow Before Sending the Contract

Olamide Olaniyan — Thu, 30 Apr 2026 19:13:00 +0000

The easiest time to catch a bad creator partnership is before legal gets involved.

That sounds obvious, but a lot of teams still do due diligence backward.

They shortlist the creator, fall in love with the idea, align internally, and only then start asking the uncomfortable questions.

By that point, people are already committed.

That is why I think creator due diligence should be treated less like casual influencer research and more like vendor review.

If you are about to send a contract, you should already know:

whether the audience looks credible
whether the content fits your brand
whether sponsored posts feel natural or forced
whether there are obvious safety or reliability risks
whether the creator can actually handle the kind of partnership you want

This post is the workflow I would build for that review process, what I would automate, and where I would still keep a human in the loop.

Why This Needs More Than a Spreadsheet

A lot of teams still run creator due diligence through:

screenshots
media kits
gut feel
one or two internal Slack messages

That is not a workflow.

That is a memory test.

Once a team is vetting more than a few creators per campaign, I want a system that creates a review packet with the same questions every time.

Consistency is the whole point.

The Review Packet I Want

Before any contract goes out, I want a packet with five sections.

1. Audience quality

Recent engagement, comment quality, follower sanity checks.

2. Content fit

Tone, niche relevance, production quality, audience expectations.

3. Sponsored-post history

How prior branded posts performed and whether they felt believable.

4. Risk scan

Recent controversies, hostile comment trends, category conflicts, operational instability.

5. Recommendation

Approve, caution, or decline.

That is the minimum useful structure.

JavaScript Version: Creator Review Packet Builder

This version keeps the workflow intentionally simple.

const headers = {
  'X-API-Key': process.env.SOCIAVAULT_API_KEY,
};

async function fetchJson(url) {
  const response = await fetch(url, { headers });
  if (!response.ok) {
    throw new Error(`Request failed with ${response.status}`);
  }
  return response.json();
}

function summarizeVideos(videos = []) {
  const recent = videos.slice(0, 12);
  const avgViews = recent.length
    ? recent.reduce((sum, video) => sum + (video.stats?.playCount || 0), 0) / recent.length
    : 0;
  const avgLikes = recent.length
    ? recent.reduce((sum, video) => sum + (video.stats?.diggCount || 0), 0) / recent.length
    : 0;

  return {
    recentCount: recent.length,
    avgViews: Math.round(avgViews),
    avgLikes: Math.round(avgLikes),
  };
}

function buildRecommendation({ suspiciousSignals, contentRiskSignals, operationalRiskSignals }) {
  const totalRisk = suspiciousSignals + contentRiskSignals + operationalRiskSignals;
  if (totalRisk >= 5) return 'decline';
  if (totalRisk >= 3) return 'caution';
  return 'approve';
}

async function buildCreatorPacket(handle) {
  const [profileJson, videosJson] = await Promise.all([
    fetchJson(`https://api.sociavault.com/v1/scrape/tiktok/profile?handle=${encodeURIComponent(handle)}`),
    fetchJson(`https://api.sociavault.com/v1/scrape/tiktok/videos?handle=${encodeURIComponent(handle)}&amount=12&sort_by=latest&trim=true`),
  ]);

  const profile = profileJson.data || {};
  const videos = videosJson.data || [];
  const summary = summarizeVideos(videos);

  const suspiciousSignals = [
    (profile.stats?.followerCount || 0) > 100000 && summary.avgLikes < 500,
    videos.length > 0 && videos.filter(video => (video.stats?.playCount || 0) < 1000).length > videos.length * 0.7,
  ].filter(Boolean).length;

  const contentRiskSignals = [
    videos.filter(video => ((video.desc || '').length < 5)).length > videos.length * 0.5,
  ].filter(Boolean).length;

  const operationalRiskSignals = [
    videos.length < 5,
  ].filter(Boolean).length;

  return {
    handle,
    audienceQuality: {
      followers: profile.stats?.followerCount || 0,
      recentContent: summary,
      suspiciousSignals,
    },
    contentFit: {
      recentCaptions: videos.slice(0, 5).map(video => video.desc || ''),
      contentRiskSignals,
    },
    operationalFit: {
      operationalRiskSignals,
      consistentRecentOutput: videos.length >= 5,
    },
    recommendation: buildRecommendation({ suspiciousSignals, contentRiskSignals, operationalRiskSignals }),
  };
}

buildCreatorPacket('creator_handle')
  .then(packet => console.log(packet))
  .catch(error => console.error(error));

This is not meant to replace judgment.

It is meant to keep judgment from being inconsistent.

That is a different and much more achievable goal.

Python Version: Easier to Use for Internal Review Lists

If you are running creator operations or internal approval scripts, Python is a comfortable fit.

import os
import requests


HEADERS = {'X-API-Key': os.environ['SOCIAVAULT_API_KEY']}


def fetch_json(url):
    response = requests.get(url, headers=HEADERS, timeout=30)
    response.raise_for_status()
    return response.json()


def summarize_videos(videos):
    recent = videos[:12]
    avg_views = sum(video.get('stats', {}).get('playCount', 0) for video in recent) / len(recent) if recent else 0
    avg_likes = sum(video.get('stats', {}).get('diggCount', 0) for video in recent) / len(recent) if recent else 0
    return {
        'recentCount': len(recent),
        'avgViews': round(avg_views),
        'avgLikes': round(avg_likes),
    }


def build_recommendation(suspicious_signals, content_risk_signals, operational_risk_signals):
    total_risk = suspicious_signals + content_risk_signals + operational_risk_signals
    if total_risk >= 5:
        return 'decline'
    if total_risk >= 3:
        return 'caution'
    return 'approve'


def build_creator_packet(handle):
    profile_json = fetch_json(f'https://api.sociavault.com/v1/scrape/tiktok/profile?handle={handle}')
    videos_json = fetch_json(f'https://api.sociavault.com/v1/scrape/tiktok/videos?handle={handle}&amount=12&sort_by=latest&trim=true')

    profile = profile_json.get('data') or {}
    videos = videos_json.get('data') or []
    summary = summarize_videos(videos)

    suspicious_signals = sum([
        (profile.get('stats', {}).get('followerCount', 0) > 100000 and summary['avgLikes'] < 500),
        (len(videos) > 0 and sum(1 for video in videos if video.get('stats', {}).get('playCount', 0) < 1000) > len(videos) * 0.7),
    ])

    content_risk_signals = sum([
        sum(1 for video in videos if len(video.get('desc') or '') < 5) > len(videos) * 0.5,
    ])

    operational_risk_signals = sum([
        len(videos) < 5,
    ])

    return {
        'handle': handle,
        'audienceQuality': {
            'followers': profile.get('stats', {}).get('followerCount', 0),
            'recentContent': summary,
            'suspiciousSignals': suspicious_signals,
        },
        'contentFit': {
            'recentCaptions': [(video.get('desc') or '') for video in videos[:5]],
            'contentRiskSignals': content_risk_signals,
        },
        'operationalFit': {
            'operationalRiskSignals': operational_risk_signals,
            'consistentRecentOutput': len(videos) >= 5,
        },
        'recommendation': build_recommendation(suspicious_signals, content_risk_signals, operational_risk_signals),
    }


print(build_creator_packet('creator_handle'))

What I Would Keep Manual

This is important.

I would not fully automate:

final brand-fit judgment
controversy review
contract-specific rights considerations
subjective review of sponsored-post quality

Those still deserve human eyes.

What I want to automate is the repetitive first-pass review.

That is what saves time without making the process sloppy.

Honest Alternatives

There are a few reasonable options here.

Spreadsheet + manual checks

Fine if you review a handful of creators once in a while.

Weak if you need consistency across campaigns.

Full influencer platform

Makes sense for larger teams.

Often heavier and more expensive than smaller teams need.

Lightweight due-diligence workflow on top of public data

This is where I like SociaVault. It gives me the public creator data layer so I can build a repeatable internal review process without turning the project into a scraping company.

That is a much better trade if the actual product is the decision workflow.

Final Take

Creator due diligence should happen before your team gets emotionally attached to the partnership.

That is the real value of a workflow like this.

Not that it replaces judgment, but that it makes judgment consistent and harder to skip.

If you want to build that kind of creator review process without wiring up the public data layer from scratch, SociaVault is a strong place to start.

Then keep the rest disciplined: audience quality, content fit, risk review, recommendation.

That is enough to make the contract stage much less messy.

webdev #python #javascript #marketing #automation

How to Detect Creative Fatigue From Ad Library Snapshots

Olamide Olaniyan — Wed, 29 Apr 2026 12:15:00 +0000

Creative fatigue is one of those problems teams usually notice too late.

The ad is still live. Spend is still flowing. Nobody wants to admit the creative has gone stale because the dashboard does not scream about it yet.

Then performance gets weird, the team starts talking about "market conditions," and suddenly everyone agrees the ad has probably been tired for weeks.

The frustrating part is that you often can see fatigue clues earlier, even from public data.

Not perfect proof. Not ROAS. Not internal frequency.

But enough signal to make better decisions.

That is what this post is about: how to use public ad library snapshots to detect creative fatigue, how I score it, what the signal can and cannot tell you, and how to build the workflow in both JavaScript and Python.

What Creative Fatigue Looks Like in Public Data

If you do not have access to the ad account, you cannot see the internal performance metrics that paid teams obsess over.

But you can still look for a few strong clues:

the same headlines show up again and again
the same CTA keeps repeating
landing pages do not meaningfully change
the creative set stays large enough to look active, but not fresh enough to look exploratory
new variants feel cosmetic instead of strategic

That is usually the beginning of fatigue.

Or at least the beginning of slowed testing.

Either way, it is worth paying attention to.

Why Snapshots Matter More Than Screenshots

One screenshot tells you an ad exists.

A sequence of snapshots tells you whether the advertiser is still learning.

That is the core idea.

Creative fatigue is not a single-state problem. It is a time problem.

If you want to catch it, you need a timeline.

That means saving structured snapshots of:

headlines
body text
CTA
landing page URL
platform
date captured

Once you have that, you can calculate things like:

creative carryover rate
novelty rate
CTA repetition
landing-page stagnation

Those are surprisingly good directional signals.

The Simple Model I Actually Use

I am not trying to build a perfect fatigue oracle.

I am trying to answer a useful question:

does this advertiser look like they are still testing, or just recycling?

That is enough.

My basic model uses four signals:

Carryover rate: how much of the current creative set also existed in the last snapshot
Novelty rate: the inverse of carryover
CTA repetition: whether new creative comes with new asks, or the same one repeated everywhere
Landing-page stagnation: whether the destination set changes at all

If carryover is high and both CTA and landing-page variety are low, I start flagging fatigue.

JavaScript Version: Snapshot + Fatigue Score

This is a straightforward version using Facebook and LinkedIn public ad data.

import fs from 'fs/promises';

const headers = { 'X-API-Key': process.env.SOCIAVAULT_API_KEY };

async function fetchJson(url) {
  const response = await fetch(url, { headers });
  if (!response.ok) {
    throw new Error(`Request failed with ${response.status}`);
  }
  return response.json();
}

function normalizeAds(items = []) {
  return (items || []).map(item => ({
    headline: item.headline || item.title || item.snapshot?.title || '',
    body: item.body || item.text || item.snapshot?.body?.markup || '',
    cta: item.cta || item.call_to_action || item.snapshot?.cta_text || '',
    url: item.url || item.landingPageUrl || item.snapshot?.link_url || '',
  }));
}

async function captureSnapshot(brand) {
  const [facebook, linkedin] = await Promise.all([
    fetchJson(
      `https://api.sociavault.com/v1/scrape/facebook-ad-library/company-ads?companyName=${encodeURIComponent(brand)}&status=ACTIVE&trim=true`
    ),
    fetchJson(
      `https://api.sociavault.com/v1/scrape/linkedin-ad-library/search?company=${encodeURIComponent(brand)}`
    ),
  ]);

  return {
    capturedAt: new Date().toISOString(),
    facebook: normalizeAds(facebook.data),
    linkedin: normalizeAds(linkedin.data),
  };
}

function signature(ad) {
  return [ad.headline, ad.body, ad.cta, ad.url]
    .map(value => (value || '').trim().toLowerCase())
    .join('|');
}

function scoreFatigue(currentAds, previousAds) {
  const currentSignatures = new Set(currentAds.map(signature));
  const previousSignatures = new Set(previousAds.map(signature));

  const repeated = [...currentSignatures].filter(sig => previousSignatures.has(sig)).length;
  const carryoverRate = currentSignatures.size ? repeated / currentSignatures.size : 0;
  const noveltyRate = 1 - carryoverRate;

  const currentCtas = new Set(currentAds.map(ad => ad.cta).filter(Boolean));
  const currentUrls = new Set(currentAds.map(ad => ad.url).filter(Boolean));

  let score = 0;
  if (carryoverRate > 0.85) score += 45;
  else if (carryoverRate > 0.70) score += 30;

  if (currentCtas.size <= 1) score += 20;
  if (currentUrls.size <= 1) score += 20;
  if (noveltyRate < 0.15) score += 15;

  let label = 'low';
  if (score >= 40) label = 'moderate';
  if (score >= 70) label = 'high';

  return {
    creativeCount: currentSignatures.size,
    repeatedCreatives: repeated,
    carryoverRate: Number(carryoverRate.toFixed(2)),
    noveltyRate: Number(noveltyRate.toFixed(2)),
    uniqueCtas: currentCtas.size,
    uniqueLandingPages: currentUrls.size,
    fatigueScore: score,
    fatigueLabel: label,
  };
}

async function loadSnapshot(path) {
  try {
    const raw = await fs.readFile(path, 'utf8');
    return JSON.parse(raw);
  } catch {
    return null;
  }
}

async function saveSnapshot(path, snapshot) {
  await fs.writeFile(path, JSON.stringify(snapshot, null, 2));
}

const path = './hubspot-fatigue.json';
const previousSnapshot = await loadSnapshot(path);
const currentSnapshot = await captureSnapshot('HubSpot');

if (previousSnapshot) {
  const currentAds = [...currentSnapshot.facebook, ...currentSnapshot.linkedin];
  const previousAds = [...previousSnapshot.facebook, ...previousSnapshot.linkedin];
  console.log(scoreFatigue(currentAds, previousAds));
}

await saveSnapshot(path, currentSnapshot);

This is not trying to guess exact ad performance.

It is trying to measure how much of the public creative surface still looks fresh.

That is a more realistic goal.

If you want to skip the collection layer and focus on the snapshot and scoring logic, that is exactly where SociaVault fits well.

Python Version: Same Signals, Different Stack

If your monitoring jobs run in Python, the same idea is easy to port.

import json
import os
from pathlib import Path
from datetime import datetime, timezone

import requests


HEADERS = {'X-API-Key': os.environ['SOCIAVAULT_API_KEY']}


def fetch_json(url):
    response = requests.get(url, headers=HEADERS, timeout=30)
    response.raise_for_status()
    return response.json()


def normalize_ads(items=None):
    items = items or []
    normalized = []
    for item in items:
        normalized.append({
            'headline': item.get('headline') or item.get('title') or item.get('snapshot', {}).get('title', ''),
            'body': item.get('body') or item.get('text') or item.get('snapshot', {}).get('body', {}).get('markup', ''),
            'cta': item.get('cta') or item.get('call_to_action') or item.get('snapshot', {}).get('cta_text', ''),
            'url': item.get('url') or item.get('landingPageUrl') or item.get('snapshot', {}).get('link_url', ''),
        })
    return normalized


def capture_snapshot(brand):
    facebook = fetch_json(
        f'https://api.sociavault.com/v1/scrape/facebook-ad-library/company-ads?companyName={brand}&status=ACTIVE&trim=true'
    )
    linkedin = fetch_json(
        f'https://api.sociavault.com/v1/scrape/linkedin-ad-library/search?company={brand}'
    )

    return {
        'capturedAt': datetime.now(timezone.utc).isoformat(),
        'facebook': normalize_ads(facebook.get('data')),
        'linkedin': normalize_ads(linkedin.get('data')),
    }


def signature(ad):
    return '|'.join([
        (ad.get('headline') or '').strip().lower(),
        (ad.get('body') or '').strip().lower(),
        (ad.get('cta') or '').strip().lower(),
        (ad.get('url') or '').strip().lower(),
    ])


def score_fatigue(current_ads, previous_ads):
    current_signatures = {signature(ad) for ad in current_ads}
    previous_signatures = {signature(ad) for ad in previous_ads}

    repeated = len(current_signatures & previous_signatures)
    carryover_rate = repeated / len(current_signatures) if current_signatures else 0
    novelty_rate = 1 - carryover_rate

    current_ctas = {ad.get('cta') for ad in current_ads if ad.get('cta')}
    current_urls = {ad.get('url') for ad in current_ads if ad.get('url')}

    score = 0
    if carryover_rate > 0.85:
        score += 45
    elif carryover_rate > 0.70:
        score += 30

    if len(current_ctas) <= 1:
        score += 20
    if len(current_urls) <= 1:
        score += 20
    if novelty_rate < 0.15:
        score += 15

    label = 'low'
    if score >= 40:
        label = 'moderate'
    if score >= 70:
        label = 'high'

    return {
        'creativeCount': len(current_signatures),
        'repeatedCreatives': repeated,
        'carryoverRate': round(carryover_rate, 2),
        'noveltyRate': round(novelty_rate, 2),
        'uniqueCtas': len(current_ctas),
        'uniqueLandingPages': len(current_urls),
        'fatigueScore': score,
        'fatigueLabel': label,
    }


path = Path('./hubspot-fatigue.json')
previous_snapshot = json.loads(path.read_text()) if path.exists() else None
current_snapshot = capture_snapshot('HubSpot')

if previous_snapshot:
    current_ads = current_snapshot['facebook'] + current_snapshot['linkedin']
    previous_ads = previous_snapshot['facebook'] + previous_snapshot['linkedin']
    print(score_fatigue(current_ads, previous_ads))

path.write_text(json.dumps(current_snapshot, indent=2))

What This Signal Is Good For

This kind of fatigue analysis is especially useful when you want to answer questions like:

Is this competitor still testing new ideas?
Are they over-leaning on one winner?
Has their category narrative stopped evolving?
Is a pricing, offer, or positioning change overdue?

That is valuable even if you never know the exact ROAS.

What This Signal Cannot Tell You

This is the important caveat.

High carryover does not automatically mean the campaign is failing.

Sometimes a "tired" public ad is still profitable enough internally that the team keeps it live on purpose.

So I do not treat fatigue as a verdict.

I treat it as a prompt.

Something like:

this advertiser looks less exploratory than they did three weeks ago. Why?

That is a useful question.

Honest Alternatives

There are a few other ways to approach this.

Performance data from your own ad account

Best if you are measuring your own fatigue.

Not useful for competitor monitoring.

Manual creative review

Still valuable, especially for strategy teams.

But it does not scale well or give you historical baselines.

Full creative clustering with embeddings

Powerful, but heavier than most teams need at first.

I would start with string-level signatures and only get more sophisticated if the workflow proves useful.

Final Take

Creative fatigue gets easier to spot once you stop treating the ad library like a gallery and start treating it like a time series.

Capture snapshots. Compare them. Measure carryover. Watch whether the CTA and landing-page set are evolving.

That alone will tell you more than most ad-library browsing ever will.

And if you want the public ad data layer without building four separate collection systems first, SociaVault is a good place to plug in.

webdev #marketing #python #javascript #analytics

How I'd Build a Competitor Pricing Change Detector From Ads and Landing Pages

Olamide Olaniyan — Tue, 28 Apr 2026 12:14:00 +0000

Competitors almost never announce pricing changes cleanly.

What usually happens is messier.

An ad that used to say "book a demo" now says "start free." A landing page introduces team pricing. A comparison page suddenly starts talking about replacing three tools. A savings claim appears where there used to be none.

By the time sales hears about it in calls, the move is already live.

That is why I like building pricing detection from public signals instead of waiting for market gossip.

This post is the workflow I would use: watch ad-library message shifts, pull the landing pages behind those ads, extract the parts that matter, and diff them over time.

It is a much better system than casually checking a pricing page once a month and pretending that counts as monitoring.

Why Ads Matter Here

When a company changes pricing, it usually changes the framing around the offer too.

That shows up as:

explicit price language
a new CTA like start free
annual savings messaging
bundles or consolidation language
more aggressive ROI framing

So ad libraries are not just creative research. They are early pricing signals.

That is what makes this workflow useful.

The Two Layers I Actually Need

I want two layers working together.

Layer 1: Ad monitoring

Catch pricing-related message shifts as soon as they show up in paid campaigns.

Layer 2: Landing-page monitoring

Confirm whether the actual page now reflects a new pricing, packaging, or CTA structure.

If you only look at the landing page, you are often late.

If you only look at the ads, you miss the underlying offer detail.

You need both.

JavaScript Version: Detect Pricing Signals From Public Ads

This version uses Facebook and LinkedIn ad data, then flags likely pricing changes.

const headers = {
  'X-API-Key': process.env.SOCIAVAULT_API_KEY,
};

async function fetchJson(url) {
  const response = await fetch(url, { headers });
  if (!response.ok) {
    throw new Error(`Request failed with ${response.status}`);
  }
  return response.json();
}

function normalizeAds(items = []) {
  return (items || []).map(item => ({
    headline: item.headline || item.title || item.snapshot?.title || '',
    body: item.body || item.text || item.snapshot?.body?.markup || '',
    cta: item.cta || item.call_to_action || item.snapshot?.cta_text || '',
    url: item.url || item.landingPageUrl || item.snapshot?.link_url || '',
  }));
}

function isPricingSignal(ad) {
  const text = `${ad.headline} ${ad.body} ${ad.cta}`.toLowerCase();
  return /(pricing|save|annual|per month|\/month|start free|bundle|replace|book a demo|talk to sales)/.test(text);
}

function summarizeSignals(ads) {
  return ads
    .filter(isPricingSignal)
    .map(ad => ({
      headline: ad.headline,
      cta: ad.cta,
      url: ad.url,
    }));
}

async function collectSignals(company) {
  const [facebookJson, linkedinJson] = await Promise.all([
    fetchJson(
      `https://api.sociavault.com/v1/scrape/facebook-ad-library/company-ads?companyName=${encodeURIComponent(company)}&status=ACTIVE&trim=true`
    ),
    fetchJson(
      `https://api.sociavault.com/v1/scrape/linkedin-ad-library/search?company=${encodeURIComponent(company)}`
    ),
  ]);

  return {
    facebook: summarizeSignals(normalizeAds(facebookJson.data)),
    linkedin: summarizeSignals(normalizeAds(linkedinJson.data)),
  };
}

collectSignals('HubSpot')
  .then(signals => console.log(signals))
  .catch(error => console.error(error));

That gets me the first layer: which ads currently look pricing-relevant.

If you want a public ad-data layer without wiring every platform separately, SociaVault is the piece I would use for that.

Python Version: Add Landing Page Extraction

Once I have the ad URLs, I want to inspect the pages too.

import os
import re
import requests


HEADERS = {'X-API-Key': os.environ['SOCIAVAULT_API_KEY']}


def fetch_json(url):
    response = requests.get(url, headers=HEADERS, timeout=30)
    response.raise_for_status()
    return response.json()


def normalize_ads(items=None):
    items = items or []
    normalized = []
    for item in items:
        normalized.append({
            'headline': item.get('headline') or item.get('title') or item.get('snapshot', {}).get('title', ''),
            'body': item.get('body') or item.get('text') or item.get('snapshot', {}).get('body', {}).get('markup', ''),
            'cta': item.get('cta') or item.get('call_to_action') or item.get('snapshot', {}).get('cta_text', ''),
            'url': item.get('url') or item.get('landingPageUrl') or item.get('snapshot', {}).get('link_url', ''),
        })
    return normalized


def is_pricing_signal(ad):
    text = f"{ad['headline']} {ad['body']} {ad['cta']}".lower()
    return any(term in text for term in ['pricing', 'save', 'annual', 'per month', '/month', 'start free', 'bundle', 'replace', 'book a demo', 'talk to sales'])


def fetch_html(url):
    response = requests.get(
        url,
        headers={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'},
        timeout=30,
    )
    response.raise_for_status()
    return response.text


def extract(html, pattern):
    match = re.search(pattern, html, re.IGNORECASE | re.DOTALL)
    return re.sub(r'\s+', ' ', match.group(1)).strip() if match else None


def parse_page(html, url):
    return {
        'url': url,
        'title': extract(html, r'<title>(.*?)</title>'),
        'h1': extract(html, r'<h1[^>]*>(.*?)</h1>'),
        'cta': extract(html, r'<a[^>]*>(Start free|Book a demo|Talk to sales|Try free|Get started)</a>'),
    }


def collect_pricing_detector(company):
    facebook = fetch_json(
        f'https://api.sociavault.com/v1/scrape/facebook-ad-library/company-ads?companyName={company}&status=ACTIVE&trim=true'
    )
    linkedin = fetch_json(
        f'https://api.sociavault.com/v1/scrape/linkedin-ad-library/search?company={company}'
    )

    ads = normalize_ads(facebook.get('data')) + normalize_ads(linkedin.get('data'))
    candidate_ads = [ad for ad in ads if is_pricing_signal(ad) and ad.get('url')]

    pages = []
    for ad in candidate_ads[:10]:
        try:
            html = fetch_html(ad['url'])
            pages.append(parse_page(html, ad['url']))
        except Exception as error:
            print(f'Failed to fetch {ad["url"]}: {error}')

    return {
        'signals': candidate_ads,
        'pages': pages,
    }


print(collect_pricing_detector('HubSpot'))

That gives me both layers:

the pricing-related ads
the current landing pages behind them

Which is exactly what I need to spot changes earlier.

What I Watch For First

The most useful signs are usually:

CTA shift from demo to self-serve
explicit price language appearing in ads
bundling or consolidation language on the page
hero headline changing from feature-led to savings-led
comparison page rollout tied to active ad campaigns

Those often tell you more than a formal pricing page refresh.

Honest Alternatives

There are a few other approaches.

Manual competitor review

Fine for one or two companies once a quarter.

Usually too inconsistent to rely on.

Pure landing-page monitoring

Helpful, but often later than the ad-side signal.

Full site crawler

Powerful, but more breadth than you need if the real interest is active paid traffic.

That is why I like starting from ad-linked pages.

They are higher signal.

Final Take

Competitor pricing changes usually show up as public clues before they show up as clean announcements.

That is why I like building a detector around ad-library signals plus landing-page diffs.

It is faster, more repeatable, and much less dependent on memory.

If you want to build that workflow without stitching together the public ad layer by hand, SociaVault is a good place to start.

Then keep the rest straightforward: collect the ads, identify pricing signals, fetch the landing pages, diff the language.

That alone gives you a very useful early-warning system.

webdev #python #javascript #marketing #competitiveanalysis