DEV Community

Erik anderson
Erik anderson

Posted on • Originally published at scanbrief.dev

I Built an AI Pipeline That Reads 20+ Tech Sources So I Don't Have To

I was drowning in tabs every morning. Hacker News, GitHub Trending, ArXiv, TechCrunch, AI lab blogs — all open, half-read, mostly redundant. The same Claude 4.5 announcement on 8 different sites. The same trending repo summarized in 3 newsletters.

Sound familiar?

So I built ScanBrief — an AI-powered intelligence pipeline that does my morning reading in 2 minutes.

What It Does

ScanBrief ingests from 20+ high-signal sources on a daily schedule:

  • Hacker News (Firebase API — top 30 stories with points/comments)
  • GitHub Trending (daily trending repos with star counts)
  • ArXiv (cs.AI, cs.LG, cs.SE)
  • AI Lab Blogs (OpenAI, Google AI, Anthropic)
  • Tech Press (TechCrunch, Ars Technica, The Verge)
  • Product Hunt, Reddit, Dev.to (coming in Sprint 1)

Then it:

  1. Deduplicates — URL hash + fuzzy title matching. Same story on HN and Reddit? You see it once.
  2. Scores — Source authority × novelty bonus. Not just popularity — actual signal quality.
  3. Summarizes — Claude AI generates 2-sentence summaries for the top 15 items.
  4. Delivers — Morning brief to Discord, email, Slack, or API.

The Brief Format

You get a clean markdown brief every morning:

  • Top 10 — The signals you absolutely should not miss
  • Also Notable — 5 more items worth a glance
  • Each item: title, source, score, 2-sentence AI summary, link

Total read time: ~5 minutes. Covers what would take 2+ hours of manual scanning.

Why I Built This (Not Another Newsletter)

Newsletters are humans picking links. That is fine, but:

  • They miss things (human bandwidth is finite)
  • They do not deduplicate across sources
  • They cannot score by YOUR relevance (what affects YOUR stack)
  • They publish on their schedule, not yours

ScanBrief is infrastructure, not editorial. It is a pipeline you own.

The Stack

  • Python — ingesters, dedup engine, scorer, brief formatter
  • Claude AI (Haiku) — summarization via CLI
  • SQLite — item storage, dedup history, brief archive
  • Discord webhooks — delivery (email + API coming)
  • Cron — 6:30 AM daily, before my morning ops check

Full pipeline runs in ~2 minutes. Without AI summaries, 6 seconds.

Pricing

Tier Price What You Get
Free /bin/bash Daily email, top 5 items, 3 tracks
Pro /mo Full brief, all tracks, emerging signals, custom sources, API
Team 9/mo Shared dashboard, team tracks, webhooks, priority support

Early subscribers get Pro free for 3 months.

What is Next

  • Sprint 1: ArXiv, Product Hunt, Reddit ingesters + track categorization
  • Sprint 2: Project relevance mapping — "this trending thing affects YOUR codebase"
  • Sprint 3: Public web dashboard + email delivery
  • Sprint 4: Stripe payments, API endpoint, BYOK option

Try It

Check it out at scanbrief.dev. Sign up for the free tier and get the brief tomorrow morning.

Signal compounds. Noise compounds too. Choose which one you are stacking.


Built by Prime Automation Solutions. I write about building autonomous systems at @primex001.

Top comments (0)