Ai tools overview

Posted on Jul 17

How to Summarize PDFs Locally with Open-Source LLMs (No API, No Data Leaving Your Machine)

#ai #python #tutorial #opensource

Most "summarize a PDF with AI" tutorials send your document to a cloud API. That's fine — until the document is a contract, a patient record, or anything your compliance team would rather not ship to a third party. The alternative in 2026 is genuinely good: run an open-source model locally, so the bytes never leave your machine and you pay $0 per token.

Here's how to build a local PDF summarizer with Ollama and Llama 3, plus an honest look at where local wins and where it doesn't.

Why local (and why not)

Local wins when:

Privacy/compliance — the file can't leave your infrastructure.
Volume — you summarize thousands of docs and don't want a per-token bill.
Offline / air-gapped environments.

Local costs you:

Hardware — you want a decent GPU (or Apple Silicon) for reasonable speed; CPU-only works but is slow.
Quality ceiling — a 7B–8B local model is weaker at long, nuanced documents than a frontier cloud model.
Ops — you own the setup, the model updates, and the tuning.

If none of those first three apply to you, a cloud API or a no-code tool is probably less hassle (more on that at the end).

Step 1: Install Ollama and pull a model

Ollama makes running local models a one-liner. After installing it:

ollama pull llama3.1:8b
# smaller/faster: ollama pull llama3.2:3b
# stronger, needs more VRAM: ollama pull llama3.1:70b

Model choice is the main quality/speed dial. 3B is fast and fine for short docs; 8B is the sweet spot for most machines; 70B approaches cloud quality if you have the VRAM.

Step 2: Extract the text

Same as any pipeline — native PDFs give text directly; scanned PDFs need OCR first.

from pypdf import PdfReader

def extract_text(path: str) -> str:
    reader = PdfReader(path)
    return "\n".join((page.extract_text() or "") for page in reader.pages)

If this comes back nearly empty, the PDF is scanned — run it through Tesseract (pytesseract) before continuing.

Step 3: Chunk, then summarize with the local model

Local models have context limits too, so long documents still need the map-reduce pattern: summarize each chunk, then summarize the summaries. The only difference from a cloud pipeline is the client — we call Ollama instead of a remote API.

# pip install ollama pypdf
import ollama

MODEL = "llama3.1:8b"

def chunk(text: str, size: int = 8000, overlap: int = 400):
    step = size - overlap
    return [text[i:i + size] for i in range(0, len(text), step)]

def summarize(text: str) -> str:
    resp = ollama.chat(
        model=MODEL,
        messages=[
            {"role": "system", "content": "Summarize the text into concise bullet points. Keep names, numbers, and conclusions. Do not invent anything."},
            {"role": "user", "content": text},
        ],
        options={"temperature": 0.2},  # lower = more faithful, less creative
    )
    return resp["message"]["content"]

def summarize_pdf(path: str) -> str:
    text = extract_text(path)
    if len(text.strip()) < 50:
        raise ValueError("Almost no text extracted — this PDF is probably scanned. OCR it first.")
    partials = [summarize(c) for c in chunk(text)]
    combined = "\n\n".join(partials)
    return summarize("Combine these section summaries into one structured summary:\n\n" + combined)

if __name__ == "__main__":
    print(summarize_pdf("report.pdf"))

Note the smaller chunk size (8k vs the 12k I'd use on a cloud model): local 8B models hold quality better on tighter chunks, and it keeps each call fast.

Step 4: Keep it faithful

Local models hallucinate more than frontier ones, so lean on the prompt and settings:

Low temperature (0.2 or below) for summaries — you want fidelity, not flair.
Explicit instruction not to invent facts, and to preserve numbers/names.
Spot-check a few outputs against the source before trusting the pipeline on a batch.

Performance reality check

On an 8B model:

Apple Silicon (M-series) / a modern GPU: a 30–50 page report summarizes in seconds to a couple of minutes.
CPU-only: it works, but expect minutes per document — fine for a nightly batch, painful interactively.

The cost, though, is the headline: after the download, summarizing 10,000 PDFs costs the same as summarizing one — electricity. That's the whole reason to go local at volume.

When local isn't the right call

Being honest about the trade-off, since this is where a lot of "run it locally!" posts stop:

You just need a few summaries occasionally. Standing up Ollama, a model, and an extraction pipeline to summarize five PDFs is overkill. If privacy isn't the constraint, a free web tool does it in seconds — ChatPDF and NotebookLM if you don't mind an account, or PDFSummarizer.net if you want no sign-up and formats like EPUB/PPTX handled for you. One caveat that matters specifically because this article is about privacy: those are hosted tools, so your file goes to their servers — they're the convenience option, not the privacy option. If keeping data local is the whole point, stay local.
You need top-tier reasoning on long, subtle documents. A frontier cloud model still edges out an 8B local one.
You don't have the hardware. CPU-only 8B is slow. Below a certain machine, cloud is simply faster and cheaper in wall-clock terms.

Takeaways

Local summarization is real in 2026: Ollama + Llama 3 gives you offline, zero-per-token summaries.
The pipeline is the same extract → chunk → map-reduce; only the model client changes.
Trade-offs: privacy and cost for hardware and a quality ceiling.
Keep temperature low and spot-check for hallucinations.
If privacy and volume aren't your drivers, a cloud API or a free no-code tool is less work.

Running models locally for document work? I'd like to hear which model/size you settled on and what hardware you're on — drop it in the comments.

Tool details were accurate at the time of writing — check current limits before you rely on them.

DEV Community