DEV Community: Techno Neighbour

A month of building Focal Harvest: what changed and where it's going published

Techno Neighbour — Thu, 30 Jul 2026 08:43:11 +0000

It's been almost a month since I first pushed Focal Harvest to GitHub, and the project looks almost nothing like its original form. New contributors, new ideas, and a lot of late nights have taken it from a scrappy personal tool into something that actually does what I originally wanted it to do.

If you haven't seen it before, here's the short version. If you have, skip ahead to the updates.

What Focal Harvest is

The problem it solves is a specific kind of tedium: you need to research something, so you open a search engine, open fifteen tabs, skim past the SEO filler, copy the parts that matter into a doc, paste it all into an LLM, and repeat the whole thing next week because something changed.

Focal Harvest automates that loop. Give it a topic and a focus area, and it searches the web, scrapes and cleans the pages concurrently, synthesises a structured Markdown report, and writes it to disk. There's a loop mode for recurring monitoring — it re-runs on a schedule and only alerts you when the content has actually changed, using MD5 fingerprinting to suppress notifications when nothing is new.

It runs on a laptop with no GPU, no Docker, no database. Reports are plain Markdown and JSON files you can open by double-clicking. It also works fully offline with no API key — the built-in extractive summarizer scores and ranks sentences by keyword density and position, producing a real report without touching any external model.

That's the base. Here's what changed in the last month.

What's new

A plugin system for site-specific parsing

The generic readability + BeautifulSoup fallback works well for standard articles. It doesn't work well for GitHub repo pages, arXiv abstracts, Reddit threads, Amazon product listings, or Stack Overflow questions — pages with structure that readability doesn't understand.

The fix was a plugin architecture. Each plugin is a single Python file with a SUPPORTED_DOMAINS list and a parse(html, url) function. When the scraper fetches a URL, it checks loaded plugins first before falling back to the generic parser. Right now, seventeen built-in plugins are covering GitHub, arXiv, Reddit, Hacker News, Stack Overflow, etc.

Adding a new one takes a single afternoon. The interface is simple by design.

A mobile bot listener

This one changed how I personally use the tool. Focal Harvest now runs a bot listener that polls Telegram and Discord for slash commands. From your phone:

/research 'topic' — runs a standard sweep and returns an Executive Summary plus PDF and Markdown attachments
/deep 'topic' — decomposes your topic into five sub-queries, searches each independently, and synthesizes across all of them
/url 'link' — deep-dives a specific URL or set of URLs
/status — returns server health, active provider, and report count
/help — command list

Authorization is handled via a whitelist of user IDs in config, so nothing runs unless it comes from you. The whole thing runs on plain requests polling — no webhook server, no framework, no open port required.

PDF and Word export

Reports now export to .pdf via ReportLab and .docx via python-docx, with proper heading styles, table rendering, and bullet formatting. Both are soft optional — if the package isn't installed, the tool warns and skips gracefully. The bot listener attaches the PDF and Markdown files directly to your Telegram or Discord message when a report completes.

Five-engine search replenishment

The original tool searched DuckDuckGo or Tavily and took whatever came back. If sources were thin or blocked, the report suffered.

Now there's an adaptive replenishment pipeline: Tavily → DuckDuckGo → Google Mobile → Bing → Brave, tried in sequence until the target source count is met. If you ask for five good sources and three get blocked, it keeps searching until it finds replacements. scrape_urls_adaptive handles the quality gate — any result under the character threshold gets dropped and the pipeline backfills automatically.

Three-tier archive fallback

For bot-blocked or JS-rendered pages, the scraper now tries three archive services in sequence: Wayback Machine (via direct redirect shortcut to avoid the rate-limited availability API), Memento, and Archive.ph. Cached entries are checked for Wayback's donation banner boilerplate before being served — poisoned cache entries force a fresh live crawl and overwrite the bad entry on disk.

Query decomposition

The /deep command uses decompose_query_locally to break a topic into sub-questions without an LLM planning call. Each sub-question gets its own search pass, results are deduplicated by URL, and everything feeds into a single synthesis pass. The report covers more angles of the topic than a single-query search can.

Source relevance filtering

Before synthesis, each scraped source is now scored for relevance against the original query and focus area. Sources below the threshold are dropped. This means the synthesiser only sees content that demonstrably relates to what you asked — less noise, better reports.

Proxy support

config.json now supports proxy_enabled, proxy_url, and proxy_list for random rotation. All outbound requests — search, scrape, notifications, archive fetches — route through the configured proxy. Credentials are masked in logs.

What's still missing

A few things I haven't gotten to:

Advanced query planning (the current decomposition is local and rule-based, not semantically aware)
Local document ingestion — PDFs, Word files alongside web sources
A web UI for users who don't want a terminal

If any of that interests you, the repo is open and the architecture is designed to be extended. The plugin system in particular is a good entry point — pick a site you scrape regularly and write a parser for it.

And if you've built something in this space, or you've hit the same tab-hoarding problem and solved it differently, I'd genuinely like to hear about it in the comments.

Why I built a CLI to automate web research instead of relying on browser tabs

Techno Neighbour — Tue, 30 Jun 2026 09:35:10 +0000

A few months ago I noticed something annoying about how I worked: I was spending more time collecting information than actually thinking about it.

The pattern was always the same. Open a search engine, open a dozen tabs, skim past the SEO filler and cookie banners, copy the paragraphs that actually mattered into a doc, paste the whole mess into an LLM and ask it to make sense of things. Then, a week later, do it again because whatever I was tracking had changed.

At some point I stopped asking "how do I do this faster" and started asking why I was doing it by hand at all.

Why the obvious answers didn't work

ChatGPT and Perplexity are fine for a single question. They're worse at the part I actually needed help with, which was repetition: running the same research loop on a schedule, keeping a record of what changed, and getting a notification when it did. Neither tool is built to sit in the background and check on a topic for you.

Plain scraping scripts have the opposite problem. They get you raw HTML, not understanding. You still have to strip out nav bars and footers by hand, and the moment you point one at a list-style page like Hacker News instead of a blog post, it falls apart.

And bookmarking is just deferring the problem. A folder of forty saved links isn't research, it's homework you haven't done yet.

I wanted something in between: automated enough to skip the tab-hoarding, but still producing something I could read and trust, not just a black-box answer.

So I built Focal Harvest

It's a modular CLI that runs the whole research loop, search, scrape, clean, synthesize, report, on its own, and stays lightweight enough to run on a laptop with no GPU and no database.

A single run looks like this: you give it a topic and a focus area (what you specifically want answered), it searches the web, pulls and cleans the pages, synthesizes a report, and writes it to disk. There's also a loop mode, so the same query can re-run every few hours and ping you on Discord or Telegram if you want to monitor something over time instead of researching it once.

How it's put together

I deliberately didn't build this as one big script. It's five files, each doing one job, called in sequence:

main.py        → terminal UI and orchestration
scraper.py     → search + concurrent crawling + HTML parsing
analyzer.py    → synthesis (AI or offline)
notifier.py    → saving reports, sending alerts
config_manager → reading/writing settings

main.py doesn't know anything about how scraping works internally, and scraper.py doesn't know anything about Discord webhooks. That separation made it much easier to add the offline summarizer later without touching the scraping code at all, and it's the kind of decision that only pays off once you try to change something six weeks in.

The parts that were actually hard

Getting clean text out of arbitrary HTML. readability-lxml is good at finding "the article" inside a page, but it assumes the page is an article. Point it at a Hacker News thread or a GitHub repo listing and it often returns almost nothing, because there's no single article body to extract. The fix was to treat readability as the first attempt, not the only one: if it returns under 200 characters of usable text, the code falls back to a structural BeautifulSoup pass that looks for <article>, <main>, or common content selectors instead. Two different parsing strategies, picked automatically based on what the page actually looks like.

Supporting three different LLM providers without three different code paths. Gemini, OpenAI, and Claude all have different request shapes, but the thing I actually cared about (send scraped context, get back a structured Markdown report) is identical across all of them. So each provider gets its own thin function that builds the right payload and hits the right endpoint, but all three feed into the same synthesize_topics router, and all three fall back to the same offline summarizer if the API call fails for any reason. The interface is the constant; the providers are interchangeable behind it.

An offline mode that's actually usable. No API key, no internet dependency on a third-party model, still get a real report. This is where most of the actual algorithm work went: extract keywords from the query and focus area, score every sentence in the scraped text by keyword density and position (earlier sentences in a paragraph, earlier paragraphs in a document, weighted higher), then deduplicate near-identical sentences before assembling the top results into a report with an executive summary and per-source findings. It's not as fluent as an LLM-written summary, but it's not nothing either, and it means the tool works the moment you clone it.

Staying out of the database trap. It would have been easy to reach for SQLite the moment I wanted history or saved searches. I didn't. Reports are timestamped Markdown and JSON files in a reports/ folder, and saved search presets live in a plain config.json. You can read everything by double-clicking it. No schema, no migrations, no ORM.

What it's not

Focal Harvest isn't trying to replace search engines or chat-based AI assistants. It automates the mechanical part, gathering and organizing information, so you spend your attention on evaluating it instead of assembling it. If you want a single deep conversational answer to one question, this is the wrong tool. If you want a repeatable, schedulable research pipeline that produces a file you can actually keep, that's the gap it's filling.

Where I'd like help

A few areas I haven't gotten to yet:

Additional search providers beyond Tavily and DuckDuckGo
A plugin system for custom parsing rules on specific sites
Recursive crawling (follow internal links to a set depth)
Incremental reports, so a recurring monitor only flags what actually changed instead of regenerating the whole thing
General performance work on the scraping layer

If any of that sounds interesting, or if you've built something in this space and have opinions about the architecture, I'd genuinely like to hear them. Issues and pull requests are open.

If you've got your own version of the tab-hoarding problem, I'd like to hear about it in the comments. What does your research loop look like, and where does it break down?