LocalLens: private document Q&A that runs 100% on your machine

My entry for the Hacktoberfest Week 2 "Side Hustle" challenge — built with open-weight models, zero API costs, and a business model I'd actually try.

The problem

Last year a freelancer friend was handed a 34-page client contract and asked to "just flag anything weird" before signing. She couldn't paste it into ChatGPT — the contract literally had a confidentiality clause forbidding exactly that. So she read all 34 pages, twice, on a Sunday.

That's the gap: millions of people handle documents they legally or ethically can't upload anywhere — client contracts, payroll sheets, patient intake forms, HR files. Their choices are manual reading or breaking confidentiality. There is no third option that costs less than a lawyer.

What I built

LocalLens — a command-line tool where you point at a document, ask a question in plain English, and get an answer in seconds. The twist: the AI model, the retrieval, and the inference engine all run on your own machine. No account. No API key. No subscription. Your document never touches a network.

python3 -m locallens ask -f contract.txt \
  -q "What is the notice period for terminating the agreement?"

Real output from the included demo (Qwen2.5-0.5B-Instruct, running on a 2-CPU virtual machine — nothing staged):

Q: What is the notice period for terminating the agreement?

The notice period for terminating the agreement is thirty (30) days.

(answered from sentence(s) [14, 17] of 32 — run with --show-context to see them)

Two things I'm proud of here. First, it cites its sources — every answer tells you which sentences it used, and --show-context prints them, so you can verify before acting on anything. Second, when the answer isn't in the document, it says so instead of inventing one:

Q: What is the penalty for late payment of invoices?

Not found in the document.

For a tool aimed at contracts, that abstention behavior isn't a nice-to-have. It's the whole product. A hallucinating contract assistant is worse than useless — it's a liability.

Who would pay for this, and why

Three personas, all real:

1. Freelancers and solo consultants reviewing client MSAs and SOWs. They already pay $15–30/month for contract tools that upload their data to someone else's cloud. A one-time $29 purchase that keeps everything local is an easy sell — privacy is the feature, not a checkbox.
2. Small HR and accounting firms answering "what does the policy say about X?" fifty times a week from employee handbooks and tax documents they can't share externally.
3. Clinic and legal admin staff triaging intake paperwork and engagement letters under confidentiality obligations.

The business model I'd actually try: open-source core (this repo, MIT), paid packaged apps — a polished desktop build with PDF/DOCX support and multi-document search at a one-time price. The open core builds trust with exactly the audience that cares about where their data goes; the packaging is what they pay for.

Bonus-points honesty: I have no first user yet — this was built for the challenge this week. The demo above is real output from the actual tool, and the limitations section below is real too.

How it works

Deliberately boring architecture, because boring ships:

• Retrieval (stdlib only): the document is split into sentences and ranked by keyword overlap with the question — stopword-filtered, with naive stemming and a tight-span tie-break so the most focused sentence wins. No embeddings, no dependencies, runs anywhere.
• Generation: the top sentences go into a prompt for Qwen2.5-0.5B-Instruct (Apache 2.0 open weights, Q4_K_M quantized), run through the llama.cpp binary (MIT). Total footprint: ~400 MB, ~20 tokens/sec on two CPU cores.
• The whole Python project has zero third-party dependencies. If you have Python 3.8 and the two downloaded files, it runs.

Two setup scripts fetch the binary and the model — both free, no accounts. From zero to first answer takes about five minutes, most of it downloading.

Why open weights made this possible

This product cannot exist on a closed API. The entire value proposition — "your document never leaves your machine" — is only credible because every layer is inspectable: the model weights are public (Qwen2.5, Apache 2.0), the inference engine is open source (llama.cpp, MIT), and my own code is MIT. A privacy tool built on a black-box API is a contradiction; built on open weights, it's a verifiable claim. Anyone can audit exactly what runs. That auditability is the moat — and it's only available because the weights are open.

Honest limitations

I'd rather tell you than have you discover them:

• The bundled model is 0.5 billion parameters — tiny on purpose, so it runs on any laptop CPU. It's reliable at lookup questions and honest abstention, but convoluted phrasing can stump it, and questions needing synthesis across distant clauses want a bigger model. Any GGUF drops in via --model.
• Retrieval is keyword-based, so pure paraphrases can be missed.
• .txt input only in v0.1 — PDF/DOCX parsing is roadmap, not reality.
• Prototype, not a hardened product. Verify answers against the cited sentences before acting on them.

Try it

Repo: https://github.com/swarnnim/locallens

./scripts/download-llamacpp.sh
./scripts/download-model.sh
python3 -m locallens ask -f sample_docs/freelance-nda.txt \
  -q "How long do confidentiality obligations survive?"

If you handle sensitive documents for a living, I'd genuinely like to know: what's the one document you wish you could ask questions about but can't upload anywhere? That's the roadmap.