It's been almost a month since I first pushed Focal Harvest to GitHub, and the project looks almost nothing like its original form. New contributors, new ideas, and a lot of late nights have taken it from a scrappy personal tool into something that actually does what I originally wanted it to do.
If you haven't seen it before, here's the short version. If you have, skip ahead to the updates.
What Focal Harvest is
The problem it solves is a specific kind of tedium: you need to research something, so you open a search engine, open fifteen tabs, skim past the SEO filler, copy the parts that matter into a doc, paste it all into an LLM, and repeat the whole thing next week because something changed.
Focal Harvest automates that loop. Give it a topic and a focus area, and it searches the web, scrapes and cleans the pages concurrently, synthesises a structured Markdown report, and writes it to disk. There's a loop mode for recurring monitoring — it re-runs on a schedule and only alerts you when the content has actually changed, using MD5 fingerprinting to suppress notifications when nothing is new.
It runs on a laptop with no GPU, no Docker, no database. Reports are plain Markdown and JSON files you can open by double-clicking. It also works fully offline with no API key — the built-in extractive summarizer scores and ranks sentences by keyword density and position, producing a real report without touching any external model.
That's the base. Here's what changed in the last month.
What's new
A plugin system for site-specific parsing
The generic readability + BeautifulSoup fallback works well for standard articles. It doesn't work well for GitHub repo pages, arXiv abstracts, Reddit threads, Amazon product listings, or Stack Overflow questions — pages with structure that readability doesn't understand.
The fix was a plugin architecture. Each plugin is a single Python file with a SUPPORTED_DOMAINS list and a parse(html, url) function. When the scraper fetches a URL, it checks loaded plugins first before falling back to the generic parser. Right now, seventeen built-in plugins are covering GitHub, arXiv, Reddit, Hacker News, Stack Overflow, etc.
Adding a new one takes a single afternoon. The interface is simple by design.
A mobile bot listener
This one changed how I personally use the tool. Focal Harvest now runs a bot listener that polls Telegram and Discord for slash commands. From your phone:
-
/research 'topic'— runs a standard sweep and returns an Executive Summary plus PDF and Markdown attachments -
/deep 'topic'— decomposes your topic into five sub-queries, searches each independently, and synthesizes across all of them -
/url 'link'— deep-dives a specific URL or set of URLs -
/status— returns server health, active provider, and report count -
/help— command list
Authorization is handled via a whitelist of user IDs in config, so nothing runs unless it comes from you. The whole thing runs on plain requests polling — no webhook server, no framework, no open port required.
PDF and Word export
Reports now export to .pdf via ReportLab and .docx via python-docx, with proper heading styles, table rendering, and bullet formatting. Both are soft optional — if the package isn't installed, the tool warns and skips gracefully. The bot listener attaches the PDF and Markdown files directly to your Telegram or Discord message when a report completes.
Five-engine search replenishment
The original tool searched DuckDuckGo or Tavily and took whatever came back. If sources were thin or blocked, the report suffered.
Now there's an adaptive replenishment pipeline: Tavily → DuckDuckGo → Google Mobile → Bing → Brave, tried in sequence until the target source count is met. If you ask for five good sources and three get blocked, it keeps searching until it finds replacements. scrape_urls_adaptive handles the quality gate — any result under the character threshold gets dropped and the pipeline backfills automatically.
Three-tier archive fallback
For bot-blocked or JS-rendered pages, the scraper now tries three archive services in sequence: Wayback Machine (via direct redirect shortcut to avoid the rate-limited availability API), Memento, and Archive.ph. Cached entries are checked for Wayback's donation banner boilerplate before being served — poisoned cache entries force a fresh live crawl and overwrite the bad entry on disk.
Query decomposition
The /deep command uses decompose_query_locally to break a topic into sub-questions without an LLM planning call. Each sub-question gets its own search pass, results are deduplicated by URL, and everything feeds into a single synthesis pass. The report covers more angles of the topic than a single-query search can.
Source relevance filtering
Before synthesis, each scraped source is now scored for relevance against the original query and focus area. Sources below the threshold are dropped. This means the synthesiser only sees content that demonstrably relates to what you asked — less noise, better reports.
Proxy support
config.json now supports proxy_enabled, proxy_url, and proxy_list for random rotation. All outbound requests — search, scrape, notifications, archive fetches — route through the configured proxy. Credentials are masked in logs.
What's still missing
A few things I haven't gotten to:
- Advanced query planning (the current decomposition is local and rule-based, not semantically aware)
- Local document ingestion — PDFs, Word files alongside web sources
- A web UI for users who don't want a terminal
If any of that interests you, the repo is open and the architecture is designed to be extended. The plugin system in particular is a good entry point — pick a site you scrape regularly and write a parser for it.
And if you've built something in this space, or you've hit the same tab-hoarding problem and solved it differently, I'd genuinely like to hear about it in the comments.
Top comments (1)
If you'd like to know more about Focal Harvest, I've created a website for it. Make sure to check out: techno-neighbour.github.io/focal-h...