How I Built an AI Research Agent That Scans 50+ Sources in 10 Minutes

#ai #automation #productivity #programming

Every morning I used to spend 2+ hours manually scanning AI news, tech blogs, and research papers. Now my Research Agent does it in 10 minutes while I sleep.

Here's exactly how I built it.

The Problem

As someone tracking AI developments daily, I was drowning in sources:

50+ news sites and blogs
Twitter/X threads
arXiv papers
GitHub trending
HackerNews front page

Manual scanning = 2+ hours every morning. Not sustainable.

The Solution: Research Agent Architecture

Cron (06:00 UTC)
    ↓
Source Scanner (web_search API)
    ↓
Content Fetcher (parallel requests)
    ↓
Relevance Filter (LLM scoring)
    ↓
Synthesis Engine (Claude)
    ↓
Research Brief (markdown)
    ↓
Notification (Telegram)

The Prompt (Copy-Paste Ready)

You are a research agent scanning AI/tech news.

TASK: Find the most important AI developments from the last 24 hours.

PROCESS:
1. Search for: [topic] + news/announcement/release
2. For each result, assess:
   - Relevance (1-10)
   - Novelty (is this actually new?)
   - Impact (who cares and why?)
3. Keep only items scoring 7+ on all criteria
4. For top 5 items, find PRIMARY sources
5. Synthesize into brief with unique angles

OUTPUT FORMAT:
## 🔥 Top Story
[headline + why it matters + unique angle]

## 📰 Other Notable
- [item]: [one-line insight]

## 🎯 Content Opportunities  
[topics that could make good articles]

What Went Wrong

Rate Limits

First version hammered APIs too fast. Fix: Added delays between requests, batch processing.

Generic Output

Early results were just summaries. Fix: Added "unique angle" requirement to prompt - what's the take nobody else has?

Paywalled Sources

Many quality sources behind paywalls. Fix: Prioritize sources with full content, use abstracts for paywalled.

Results

Time saved: 2 hours → 10 minutes
Coverage: More comprehensive (50+ sources vs my manual 10-15)
Quality: Better - agent finds things I'd miss
Consistency: Runs every day at 06:00, no exceptions

Key Insight

LLMs are much better at fresh attempts than repairs.

When output quality is low, don't ask it to "fix" the output. Restart with better constraints. The agent that produces this brief was rewritten 4 times before it worked well.