DEV Community

S Gr
S Gr

Posted on

How to Build a Self-Updating Content Aggregator Using Claude API and GitHub Actions

#python #ai #automation #tutorial

How to Build a Self-Updating Content Aggregator Using Claude API and GitHub Actions

Disclosure: This article contains an affiliate link to a tool I've used. You don't need it to follow this tutorial — all steps work with free tools.

Why This Matters

Content curation is valuable, but manually updating collections is tedious. In this tutorial, you'll build an automated system that fetches content from RSS feeds, uses AI to summarize and categorize it, then publishes updates to a static site — all without a server.

This is a genuine digital product foundation: once built, it runs autonomously and can be monetized through ads, affiliate links, or premium tiers.

What You'll Build

A GitHub repository that:

  • Runs daily via GitHub Actions (free)
  • Fetches content from RSS feeds you specify
  • Uses Claude API to generate summaries and extract key topics
  • Commits updates to a JSON file
  • Deploys automatically to GitHub Pages

Total cost: ~$2-5/month for API calls (or less with caching).

Prerequisites

  • GitHub account (free)
  • Anthropic API key (free tier available at console.anthropic.com)
  • Basic familiarity with JSON and YAML

Step 1: Set Up Your Repository

Create a new GitHub repository with these files:

/
├── .github/workflows/update.yml
├── script.py
├── feeds.json
├── index.html
└── output/
    └── content.json
Enter fullscreen mode Exit fullscreen mode

In feeds.json, list your RSS sources:

{
  "feeds": [
    "https://dev.to/feed",
    "https://news.ycombinator.com/rss"
  ]
}
Enter fullscreen mode Exit fullscreen mode

Step 2: Create the Python Script

Create script.py:

import feedparser
import json
import os
from anthropic import Anthropic

client = Anthropic(api_key=os.environ['ANTHROPIC_API_KEY'])

def fetch_and_process():
    with open('feeds.json') as f:
        feeds = json.load(f)['feeds']

    items = []
    for feed_url in feeds:
        feed = feedparser.parse(feed_url)
        for entry in feed.entries[:5]:  # Limit to 5 per feed
            summary = summarize_with_claude(entry.title, entry.get('summary', ''))
            items.append({
                'title': entry.title,
                'link': entry.link,
                'ai_summary': summary,
                'published': entry.get('published', '')
            })

    with open('output/content.json', 'w') as f:
        json.dump(items, f, indent=2)

def summarize_with_claude(title, content):
    message = client.messages.create(
        model="claude-3-haiku-20240307",
        max_tokens=100,
        messages=[{
            "role": "user",
            "content": f"Summarize in one sentence: {title}. {content[:500]}"
        }]
    )
    return message.content[0].text

if __name__ == '__main__':
    fetch_and_process()
Enter fullscreen mode Exit fullscreen mode

Step 3: Configure GitHub Actions

Create .github/workflows/update.yml:

name: Update Content

on:
  schedule:
    - cron: '0 8 * * *'  # Daily at 8 AM UTC
  workflow_dispatch:  # Manual trigger

jobs:
  update:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - run: |
          pip install feedparser anthropic
          python script.py
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      - uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: 'Auto-update content'
Enter fullscreen mode Exit fullscreen mode

Add your Anthropic API key to repository secrets: Settings → Secrets → Actions → New repository secret (name it ANTHROPIC_API_KEY).

Step 4: Create the Frontend

Create a simple index.html:

<!DOCTYPE html>
<html>
<head>
    <title>AI Curated News</title>
</head>
<body>
    <h1>Latest Updates</h1>
    <div id="content"></div>
    <script>
        fetch('output/content.json')
            .then(r => r.json())
            .then(items => {
                const html = items.map(item => `
                    <article>
                        <h2><a href="${item.link}">${item.title}</a></h2>
                        <p>${item.ai_summary}</p>
                        <small>${item.published}</small>
                    </article>
                `).join('');
                document.getElementById('content').innerHTML = html;
            });
    </script>
</body>
</html>
Enter fullscreen mode Exit fullscreen mode

Enable GitHub Pages: Settings → Pages → Source: main branch.

Step 5: Optimization and Scaling

To reduce API costs:

  • Cache summaries in a separate JSON file keyed by content hash
  • Use Claude Haiku (cheapest model) for summaries
  • Only process new entries by tracking processed URLs

When I was setting up email sequences to notify subscribers about updates, a tool called Perpetual Income 365 helped streamline the autoresponder configuration with pre-built templates. It's optional — you can use any email service, but it saved me setup time.

Monetization Options

  1. Affiliate links: Add relevant product recommendations to summaries
  2. Premium tier: Offer real-time updates or niche-specific feeds for $5-10/month
  3. Sponsorships: Once you have traffic, sell weekly sponsor spots
  4. API access: Let others query your curated content via API

Common Issues

GitHub Actions not running: Check your workflow file indentation (YAML is sensitive).

API rate limits: Add time.sleep(1) between API calls or implement exponential backoff.

Stale content: Increase feed limit or add more diverse sources.

Next Steps

You now have a fully automated content system. To expand:

  • Add sentiment analysis to flag trending topics
  • Create category-specific pages
  • Build a Telegram bot that posts updates
  • Use webhooks to trigger updates on new content instead of cron schedules

The key insight: automation that provides genuine value scales better than manual work. Start small, validate with real users, then expand.

Tool mentioned (affiliate link): https://breeze760.perpetualinc.hop.clickbank.net/?tid=devtohowtobuildse

Top comments (0)