DEV Community

Cover image for Build a Real-Time Trending Content Aggregator Across TikTok and YouTube
Olamide Olaniyan
Olamide Olaniyan

Posted on

Build a Real-Time Trending Content Aggregator Across TikTok and YouTube

Trending content waits for nobody. A sound goes viral on TikTok. A video format explodes on YouTube Shorts. By the time you see it on your feed, early movers already have millions of views.

What if you had a dashboard that caught trends before they hit mainstream? Across platforms, in real time, filtered to your niche?

That's what we're building — a Trending Content Aggregator that:

  1. Pulls trending videos from TikTok and YouTube simultaneously
  2. Tracks velocity to identify emerging trends vs. established ones
  3. Cross-references trends across platforms (what's trending on both?)
  4. Filters by niche and sends alerts for relevant opportunities

Why Cross-Platform Trend Detection Matters

Here's what most creators get wrong: they monitor trends on one platform.

But trends don't respect platform boundaries:

  • A dance trend starts on TikTok, hits YouTube Shorts 3-5 days later, then Instagram Reels
  • A recipe format that works on YouTube often works on TikTok with shorter cuts
  • Trending sounds on TikTok become trending audio on Instagram within a week

If you can spot a trend on Platform A before it crosses to Platform B, you have a massive first-mover advantage on Platform B.

The Stack

  • Node.js: Runtime
  • SociaVault API: TikTok trending, TikTok popular, YouTube Shorts trending, TikTok hashtags
  • better-sqlite3: Trend tracking over time
  • OpenAI: Trend classification and opportunity scoring

Step 1: Setup

mkdir trend-aggregator
cd trend-aggregator
npm init -y
npm install axios better-sqlite3 openai dotenv
Enter fullscreen mode Exit fullscreen mode

Create .env:

SOCIAVAULT_API_KEY=your_key_here
OPENAI_API_KEY=your_openai_key
Enter fullscreen mode Exit fullscreen mode

Step 2: Trend Tracking Database

Create db.js:

const Database = require('better-sqlite3');

const db = new Database('trends.db');

db.exec(`
  CREATE TABLE IF NOT EXISTS trending_videos (
    id TEXT,
    platform TEXT,
    title TEXT,
    author TEXT,
    views INTEGER DEFAULT 0,
    likes INTEGER DEFAULT 0,
    shares INTEGER DEFAULT 0,
    comments INTEGER DEFAULT 0,
    hashtags TEXT,
    sound TEXT,
    url TEXT,
    first_seen TEXT DEFAULT (datetime('now')),
    last_seen TEXT DEFAULT (datetime('now')),
    times_seen INTEGER DEFAULT 1,
    peak_position INTEGER,
    PRIMARY KEY (id, platform)
  );

  CREATE TABLE IF NOT EXISTS trending_hashtags (
    tag TEXT,
    platform TEXT,
    view_count INTEGER DEFAULT 0,
    video_count INTEGER DEFAULT 0,
    first_seen TEXT DEFAULT (datetime('now')),
    last_seen TEXT DEFAULT (datetime('now')),
    times_seen INTEGER DEFAULT 1,
    PRIMARY KEY (tag, platform)
  );

  CREATE TABLE IF NOT EXISTS trend_snapshots (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    platform TEXT,
    snapshot_type TEXT,
    data TEXT,
    created_at TEXT DEFAULT (datetime('now'))
  );

  CREATE TABLE IF NOT EXISTS cross_platform_trends (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    trend_name TEXT,
    platforms TEXT,
    first_platform TEXT,
    detected_at TEXT DEFAULT (datetime('now')),
    velocity_score REAL DEFAULT 0
  );
`);

module.exports = db;
Enter fullscreen mode Exit fullscreen mode

Step 3: Multi-Platform Trend Fetchers

Create aggregator.js:

require('dotenv').config();
const axios = require('axios');
const db = require('./db');
const OpenAI = require('openai');

const API_BASE = 'https://api.sociavault.com';
const headers = { 'Authorization': `Bearer ${process.env.SOCIAVAULT_API_KEY}` };
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

// ─── TikTok Trending ───

async function fetchTikTokTrending() {
  console.log('📱 Fetching TikTok trending...');

  const { data } = await axios.get(
    `${API_BASE}/v1/scrape/tiktok/trending`,
    { headers }
  );

  const videos = data.data?.videos || data.data || [];
  console.log(`  Found ${videos.length} trending videos`);

  storeVideos(videos, 'tiktok');
  return videos;
}

async function fetchTikTokPopular() {
  console.log('📱 Fetching TikTok popular videos...');

  const { data } = await axios.get(
    `${API_BASE}/v1/scrape/tiktok/videos/popular`,
    { headers }
  );

  const videos = data.data?.videos || data.data || [];
  console.log(`  Found ${videos.length} popular videos`);

  storeVideos(videos, 'tiktok');
  return videos;
}

async function fetchTikTokPopularHashtags() {
  console.log('#️⃣  Fetching TikTok trending hashtags...');

  const { data } = await axios.get(
    `${API_BASE}/v1/scrape/tiktok/hashtags/popular`,
    { headers }
  );

  const hashtags = data.data?.hashtags || data.data || [];
  console.log(`  Found ${hashtags.length} trending hashtags`);

  const upsert = db.prepare(`
    INSERT INTO trending_hashtags (tag, platform, view_count, video_count)
    VALUES (?, 'tiktok', ?, ?)
    ON CONFLICT(tag, platform) DO UPDATE SET
      last_seen = datetime('now'),
      times_seen = times_seen + 1,
      view_count = MAX(view_count, ?),
      video_count = MAX(video_count, ?)
  `);

  const tx = db.transaction(() => {
    for (const h of hashtags) {
      const views = h.viewCount || h.views || 0;
      const vids = h.videoCount || h.videos || 0;
      upsert.run(h.name || h.hashtag || h.title, views, vids, views, vids);
    }
  });
  tx();

  return hashtags;
}

// ─── YouTube Shorts Trending ───

async function fetchYouTubeTrending() {
  console.log('▶️  Fetching YouTube Shorts trending...');

  const { data } = await axios.get(
    `${API_BASE}/v1/scrape/youtube/shorts/trending`,
    { headers }
  );

  const videos = data.data?.videos || data.data || [];
  console.log(`  Found ${videos.length} trending Shorts`);

  storeVideos(videos, 'youtube');
  return videos;
}
Enter fullscreen mode Exit fullscreen mode

Step 4: Unified Video Storage

function storeVideos(videos, platform) {
  const upsert = db.prepare(`
    INSERT INTO trending_videos (id, platform, title, author, views, likes, shares, comments, hashtags, sound, url, peak_position)
    VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
    ON CONFLICT(id, platform) DO UPDATE SET
      last_seen = datetime('now'),
      times_seen = times_seen + 1,
      views = MAX(views, ?),
      likes = MAX(likes, ?),
      peak_position = MIN(COALESCE(peak_position, 999), ?)
  `);

  const tx = db.transaction(() => {
    videos.forEach((video, index) => {
      const id = video.id || video.videoId || video.shortId || `v_${Math.random().toString(36).slice(2)}`;
      const title = video.title || video.description || video.text || '';
      const author = video.author || video.authorName || video.channelName || video.creator || '';
      const views = video.views || video.viewCount || video.playCount || 0;
      const likes = video.likes || video.likeCount || video.diggCount || 0;
      const shares = video.shares || video.shareCount || 0;
      const comments = video.comments || video.commentCount || 0;

      const hashtags = (video.hashtags || []).map(h => 
        typeof h === 'string' ? h : h.name || h.title
      ).join(',');

      const sound = video.sound || video.music || video.audio || '';
      const soundName = typeof sound === 'string' ? sound : sound.title || sound.name || '';
      const url = video.url || video.videoUrl || '';

      upsert.run(
        id, platform, title.substring(0, 500), author,
        views, likes, shares, comments,
        hashtags, soundName, url, index + 1,
        views, likes, index + 1
      );
    });
  });

  tx();
}
Enter fullscreen mode Exit fullscreen mode

Step 5: Trend Velocity Calculator

Measure how fast a trend is accelerating:

function calculateVelocity() {
  // Videos that have appeared multiple times with growing views
  const accelerating = db.prepare(`
    SELECT *, 
      (views * 1.0 / MAX(times_seen, 1)) as views_per_appearance,
      times_seen,
      CAST((julianday('now') - julianday(first_seen)) * 24 AS INTEGER) as hours_tracked
    FROM trending_videos
    WHERE times_seen >= 2
    ORDER BY views_per_appearance DESC
    LIMIT 50
  `).all();

  // Hashtags appearing with growing engagement
  const risingHashtags = db.prepare(`
    SELECT *,
      (view_count * 1.0 / MAX(times_seen, 1)) as views_per_check,
      times_seen,
      CAST((julianday('now') - julianday(first_seen)) * 24 AS INTEGER) as hours_tracked
    FROM trending_hashtags
    WHERE times_seen >= 2
    ORDER BY views_per_check DESC
    LIMIT 30
  `).all();

  return { accelerating, risingHashtags };
}
Enter fullscreen mode Exit fullscreen mode

Step 6: Cross-Platform Trend Detection

Find trends appearing on BOTH platforms:

function detectCrossPlatformTrends() {
  console.log('\n🔄 Detecting cross-platform trends...\n');

  // Get hashtags from both platforms
  const tiktokTags = db.prepare(
    "SELECT tag FROM trending_hashtags WHERE platform = 'tiktok'"
  ).all().map(r => r.tag.toLowerCase());

  const youtubeTitles = db.prepare(
    "SELECT title, hashtags FROM trending_videos WHERE platform = 'youtube'"
  ).all();

  // Extract keywords from YouTube titles
  const youtubeKeywords = new Set();
  youtubeTitles.forEach(v => {
    const words = (v.title + ' ' + (v.hashtags || '')).toLowerCase().split(/\s+/);
    words
      .filter(w => w.length > 3 && !['this', 'that', 'with', 'from', 'have', 'will'].includes(w))
      .forEach(w => youtubeKeywords.add(w.replace(/[^a-z0-9]/g, '')));
  });

  // Find overlaps
  const crossTrends = [];

  for (const tag of tiktokTags) {
    const cleanTag = tag.replace(/[^a-z0-9]/g, '').toLowerCase();

    if (youtubeKeywords.has(cleanTag) && cleanTag.length > 4) {
      const tiktokData = db.prepare(
        "SELECT * FROM trending_hashtags WHERE LOWER(tag) = ? AND platform = 'tiktok'"
      ).get(tag);

      crossTrends.push({
        trend: tag,
        tiktokViews: tiktokData?.view_count || 0,
        platforms: ['tiktok', 'youtube'],
      });
    }
  }

  // Also check for matching sounds/titles
  const tiktokSounds = db.prepare(
    "SELECT DISTINCT sound FROM trending_videos WHERE platform = 'tiktok' AND sound != ''"
  ).all().map(r => r.sound.toLowerCase());

  const ytSounds = db.prepare(
    "SELECT DISTINCT sound, title FROM trending_videos WHERE platform = 'youtube' AND (sound != '' OR title != '')"
  ).all();

  for (const tiktokSound of tiktokSounds) {
    for (const yt of ytSounds) {
      if (
        tiktokSound.length > 5 &&
        (yt.sound?.toLowerCase().includes(tiktokSound) ||
         yt.title?.toLowerCase().includes(tiktokSound))
      ) {
        crossTrends.push({
          trend: tiktokSound,
          type: 'sound',
          platforms: ['tiktok', 'youtube'],
        });
        break;
      }
    }
  }

  if (crossTrends.length > 0) {
    console.log(`  Found ${crossTrends.length} cross-platform trends:\n`);
    crossTrends.forEach((ct, i) => {
      console.log(`  ${i + 1}. "${ct.trend}" (${ct.platforms.join(' + ')})`);
      if (ct.tiktokViews) console.log(`     TikTok views: ${ct.tiktokViews.toLocaleString()}`);
    });
  } else {
    console.log('  No cross-platform overlaps detected yet. Run more checks to build data.');
  }

  return crossTrends;
}
Enter fullscreen mode Exit fullscreen mode

Step 7: AI Trend Analysis & Opportunity Scoring

async function analyzeTrendOpportunities(niche = null) {
  const { accelerating, risingHashtags } = calculateVelocity();
  const crossTrends = detectCrossPlatformTrends();

  const nicheFilter = niche ? `for the ${niche} niche` : '';

  console.log(`\n🧠 AI trend analysis ${nicheFilter}...\n`);

  const trendData = {
    accelerating_videos: accelerating.slice(0, 15).map(v => ({
      platform: v.platform,
      title: v.title?.substring(0, 100),
      author: v.author,
      views: v.views,
      sound: v.sound,
      hashtags: v.hashtags,
      velocity: Math.round(v.views_per_appearance),
      hours_tracked: v.hours_tracked,
    })),
    rising_hashtags: risingHashtags.slice(0, 10).map(h => ({
      tag: h.tag,
      platform: h.platform,
      views: h.view_count,
      appearances: h.times_seen,
    })),
    cross_platform: crossTrends.slice(0, 10),
  };

  const completion = await openai.chat.completions.create({
    model: 'gpt-4o-mini',
    messages: [{
      role: 'user',
      content: `Analyze these trending content signals and identify the best opportunities ${nicheFilter}.

Trend Data:
${JSON.stringify(trendData, null, 2)}

Return JSON:
{
  "top_opportunities": [
    {
      "trend": "trend name/description",
      "platform": "where it's trending",
      "urgency": "high/medium/low",
      "content_idea": "specific video idea to capitalize on this",
      "why_now": "why this is an opportunity right now",
      "estimated_window": "how long this opportunity lasts"
    }
  ],
  "emerging_themes": ["broader themes emerging across platforms"],
  "cross_platform_plays": [
    {
      "trend": "what's crossing over",
      "originated_on": "platform",
      "opportunity_on": "platform where it hasn't peaked",
      "action": "specific content to create"
    }
  ],
  "sounds_to_watch": ["trending sounds worth bookmarking"],
  "hashtags_to_use": ["hashtags gaining momentum"],
  "avoid": ["trends that are already oversaturated"]
}`
    }],
    response_format: { type: 'json_object' }
  });

  const analysis = JSON.parse(completion.choices[0].message.content);

  // Print report
  console.log('🔥 TREND INTELLIGENCE REPORT');
  console.log(''.repeat(60));

  console.log('\n🎯 TOP OPPORTUNITIES:');
  console.log(''.repeat(50));
  (analysis.top_opportunities || []).forEach((opp, i) => {
    const urgencyIcon = opp.urgency === 'high' ? '🔴' : opp.urgency === 'medium' ? '🟡' : '🟢';
    console.log(`\n  ${urgencyIcon} ${i + 1}. ${opp.trend}`);
    console.log(`     Platform: ${opp.platform}`);
    console.log(`     Idea: ${opp.content_idea}`);
    console.log(`     Why now: ${opp.why_now}`);
    console.log(`     Window: ${opp.estimated_window}`);
  });

  if (analysis.cross_platform_plays?.length > 0) {
    console.log('\n\n🔄 CROSS-PLATFORM PLAYS:');
    console.log(''.repeat(50));
    analysis.cross_platform_plays.forEach((cp, i) => {
      console.log(`\n  ${i + 1}. "${cp.trend}"`);
      console.log(`     ${cp.originated_on}${cp.opportunity_on}`);
      console.log(`     Action: ${cp.action}`);
    });
  }

  console.log('\n\n📌 Hashtags to use:');
  (analysis.hashtags_to_use || []).forEach(h => console.log(`  #${h}`));

  console.log('\n🎵 Sounds to watch:');
  (analysis.sounds_to_watch || []).forEach(s => console.log(`  🔊 ${s}`));

  console.log('\n⚠️  Avoid (oversaturated):');
  (analysis.avoid || []).forEach(a => console.log(`  ✗ ${a}`));

  return analysis;
}
Enter fullscreen mode Exit fullscreen mode

Step 8: Automated Polling

async function poll(intervalMinutes = 60) {
  console.log(`🔄 Starting trend polling every ${intervalMinutes} minutes...`);
  console.log('   Press Ctrl+C to stop.\n');

  async function runCheck() {
    const timestamp = new Date().toLocaleTimeString();
    console.log(`\n${''.repeat(60)}`);
    console.log(`⏰ Check at ${timestamp}`);
    console.log(`${''.repeat(60)}\n`);

    try {
      await fetchTikTokTrending();
      await new Promise(r => setTimeout(r, 1500));

      await fetchTikTokPopular();
      await new Promise(r => setTimeout(r, 1500));

      await fetchTikTokPopularHashtags();
      await new Promise(r => setTimeout(r, 1500));

      await fetchYouTubeTrending();

      detectCrossPlatformTrends();

      const { accelerating } = calculateVelocity();
      if (accelerating.length > 0) {
        console.log(`\n⚡ ${accelerating.length} accelerating trends detected`);
      }
    } catch (err) {
      console.error(`  Error: ${err.message}`);
    }
  }

  await runCheck();
  setInterval(runCheck, intervalMinutes * 60 * 1000);
}
Enter fullscreen mode Exit fullscreen mode

Step 9: CLI

async function main() {
  const command = process.argv[2];

  switch (command) {
    case 'check':
      await fetchTikTokTrending();
      await fetchTikTokPopular();
      await fetchTikTokPopularHashtags();
      await fetchYouTubeTrending();
      console.log('\n✅ All platforms checked.');
      break;

    case 'analyze':
      const niche = process.argv[3];
      await analyzeTrendOpportunities(niche);
      break;

    case 'cross':
      detectCrossPlatformTrends();
      break;

    case 'velocity':
      const { accelerating, risingHashtags } = calculateVelocity();
      console.log(`\n⚡ ${accelerating.length} accelerating videos:`);
      accelerating.slice(0, 10).forEach((v, i) => {
        console.log(`  ${i+1}. [${v.platform}] ${v.title?.substring(0, 60)} (${v.views.toLocaleString()} views, seen ${v.times_seen}x)`);
      });
      console.log(`\n#️⃣  ${risingHashtags.length} rising hashtags:`);
      risingHashtags.slice(0, 10).forEach((h, i) => {
        console.log(`  ${i+1}. #${h.tag} (${h.view_count.toLocaleString()} views, seen ${h.times_seen}x)`);
      });
      break;

    case 'poll':
      const interval = parseInt(process.argv[3]) || 60;
      await poll(interval);
      break;

    default:
      console.log('Trending Content Aggregator\n');
      console.log('Commands:');
      console.log('  node aggregator.js check                - Fetch all platforms');
      console.log('  node aggregator.js analyze [niche]       - AI opportunity analysis');
      console.log('  node aggregator.js cross                 - Cross-platform trends');
      console.log('  node aggregator.js velocity              - Show accelerating trends');
      console.log('  node aggregator.js poll [minutes]        - Auto-poll (default: 60min)');
  }
}

main().catch(console.error);
Enter fullscreen mode Exit fullscreen mode

Running It

# One-time check across all platforms
node aggregator.js check

# AI analysis for your niche
node aggregator.js analyze "fitness"

# Check for cross-platform trends
node aggregator.js cross

# Auto-poll every 30 minutes
node aggregator.js poll 30
Enter fullscreen mode Exit fullscreen mode

The Early Mover Advantage

Here's why timing matters so much with trends:

Timing Views Potential
First 12 hours 10x–100x normal
Day 1-3 5x–20x normal
Day 4-7 2x–5x normal
After week 1 Normal or below (oversaturated)

By the time you see a trend on your For You page, you're in the "day 4-7" window at best. This tool puts you in the "first 12 hours" window.

Cost Comparison

Tool Monthly Cost Platforms
TrendTok $9.99/mo TikTok only
Exploding Topics $39/mo Web trends (not social)
Sprout Social $249/mo Listening, not trend detection
VidIQ $7.50/mo YouTube only
This tool ~$0.10/check cycle TikTok + YouTube + cross-platform

Get Started

  1. Get your API key at sociavault.com
  2. Run your first check: node aggregator.js check
  3. Set up hourly polling for your niche
  4. Create content within 24 hours of trend detection

The algorithm rewards early movers. Be the first, not the fiftieth.


Trends don't wait. By the time it's on your feed, it's too late. Catch them at the source.

javascript #tiktok #youtube #webdev

Top comments (0)