DEV Community

Sumit Nemade
Sumit Nemade

Posted on

LongLoom: Stop Losing Facts When Your Documents Exceed the Context Window

#opensource #ai #llm

I built an open-source library that processes documents of any length through LLM map/merge/repair — with measured, reproducible proof that no facts are lost. Here's why I built it, how it works, and what the benchmarks show.

The problem nobody talks about honestly

Every LLM has a context window. Every enterprise has documents that blow past it.

The standard advice is "just chunk it and summarize each piece." That works — until you actually measure what's lost. We did, with a planted-fact test: seed a document with known identifiers, numbers, and dates, then check how many survive a naive chunk-summarize pipeline.

The result: naive summarization drops 2–10% of facts. Different facts every run. A contract ID here, a dollar amount there. Gone. And because it's stochastic, you can't even predict what disappears.

For a chatbot, 8% fact loss is annoying. For a compliance pipeline, a legal discovery, or a financial audit — it's disqualifying.

The fix: stop asking the LLM to summarize

LongLoom's core insight is simple: the pipeline should transport information losslessly, and the LLM should only condense when you explicitly allow it.

The library splits into two modes:

Mode What the LLM does Fact preservation Use when
Summarizing (default) Reads each chunk, produces a summary Measured 2–10% loss Readable reports, chat contexts
Verbatim extraction (for_extraction()) Copies fact-bearing lines verbatim 100.0% at scale (4,367/4,367 planted facts) Compliance, legal, audit, search indexing

The verbatim mode works because the extraction prompt tells the LLM to copy lines containing facts rather than paraphrase them. The merge stage deduplicates exact copies. The result: the pipeline adds zero information loss — verified by a deterministic fake LLM that provably echoes its input, plus real NVIDIA Nemotron and OpenAI GPT-4o-mini runs.

How it works 

big document
     │
     ▼
CHUNK ──► line-aware splitting, persisted to your storage
     │
     ▼
MAP ──── one LLM call per chunk (parallel, checkpointed)
     │
     ▼
MERGE ── hierarchical combine, window-sized groups
     │
     ▼
REPAIR ─ bounded condensation, only if still too big
     │
     ▼
final result + verification report
Enter fullscreen mode Exit fullscreen mode

The context window only ever sees chunks and compact intermediate results. The document never has to fit.

It works with what you already have

Any LLM. Not a wrapper around one vendor — a dependency-injection seam:

from longloom import init_llm

# 9 provider integrations, or bring your own:
llm = init_llm("nvidia", "nvidia/nemotron-3-ultra-550b-a55b", api_key=..., thinking=False)
llm = init_llm("openai", "gpt-4o-mini", api_key=...)
llm = init_llm("anthropic", "claude-sonnet-4-20250514", api_key=...)
llm = init_llm("custom", model="my-model", base_url="http://vllm.internal:8000/v1",
               headers={"X-Tenant": "acme"})     # vLLM, LiteLLM, Azure APIM — anything OpenAI-compatible
llm = lambda prompt: my_gateway(prompt)          # even a bare function
Enter fullscreen mode Exit fullscreen mode

Any storage. Your databases, your schema, your tables:

from longloom import create_store

store = create_store("postgresql://app@host/mydb",
                     table="app_chunks",           # your table
                     columns={"id": "chunk_pk"},   # your column names
                     namespace="legal-team")       # multi-tenant safe
Enter fullscreen mode Exit fullscreen mode

9 storage targets: PostgreSQL, MySQL, MSSQL, Oracle, MongoDB, Supabase, SQLite, JSONL, in-memory. Or subclass one ABC with four methods for anything else.

What the benchmarks show

All benchmarks are reproducible — the benchmark script ships in the repo:

python examples/20_benchmark.py --sections 50000
Enter fullscreen mode Exit fullscreen mode
Benchmark Result
Pipeline losslessness 83,023/83,023 facts preserved at 6.7M chars / 2.16M tokens / 3,795 chunks — 100.0%
Parallel map speedup ≈4× (measured with real NVIDIA API, 8 workers)
Real NVIDIA extraction (91.7%) nvidia/nemotron-3-ultra at scale, verbatim mode
Real OpenAI extraction gpt-4o-mini, same pipeline, same verbatim mode
CI coverage 100% line coverage, gated at 99% in CI on Python 3.10–3.13

The 83,023-fact stress test processes a document 265× larger than the NVIDIA context window — and preserves every single fact through chunk → map → hierarchical merge → repair.

Enterprise-ready, not enterprise-themed

The things security reviews actually check:

  • Explicit credentials — api_key= is a constructor argument. The library contains zero os.environ reads for secrets (verified by test).
  • Zero telemetry — no phone-home, no analytics, no update pings.
  • Your databases — chunks live in your tables, your columns, your namespaces. No dedicated database required.
  • Injection-validated — table and column names are regex-validated before any query is built.
  • XSS-hardened — generated HTML viewers escape content and disable script execution.
  • Structured logging — longloom.* stdlib loggers; prompts, completions, and credentials are never logged (regression-tested).
  • SECURITY.md — data-flow disclosure, private vulnerability reporting.
  • CI — pytest matrix on Python 3.10–3.13, pip-audit supply-chain scan, 100% line coverage gate, MongoDB service container for live integration tests.

Honest limitations

  • Map/merge aggregates; it doesn't reason globally. Cross-chunk conflicts are preserved side by side, not reconciled.
  • Real-LLM verbatim extraction achieves ~92% fact preservation at scale (the pipeline adds zero loss; the model's quality is the model's quality).
  • The map phase is sequential per worker; parallelize via max_workers for large corpora.
  • tiktoken approximates non-OpenAI tokenizers — keep 2× headroom on chunk sizes.

Try it 

pip install longloom
Enter fullscreen mode Exit fullscreen mode 
from longloom import MapReduceProcessor, SQLiteContextStore, init_llm, PromptTemplates

processor = MapReduceProcessor(
    llm=init_llm("nvidia", "nvidia/nemotron-3-ultra-550b-a55b",
                 api_key=os.environ["NVIDIA_API_KEY"], thinking=False),
    store=SQLiteContextStore("ctx.db"),
    prompts=PromptTemplates.for_extraction(),
    max_workers=4,
)
result = processor.process(your_500_page_document, task="Extract all obligations")
Enter fullscreen mode Exit fullscreen mode

MIT licensed. 88 tests. Zero telemetry. Star it if useful — contributions welcome.

LongLoom is built on LangGraph and langchain-core. It works with NVIDIA Nemotron, OpenAI GPT, Anthropic Claude, Groq, Ollama, Mistral, DeepSeek, Together AI, and any OpenAI-compatible endpoint.

Top comments (0)