Implementing docling-context!
Introduction
Large Language Models excel at reasoning, but enterprise retrieval-augmented generation (RAG) and document processing workflows often flounder when dealing with complex, multi-modal documents. Unstructured files (such as PDFs, Markdown, and Office formats) frequently suffer from structural loss during chunking, leading to imprecise context window allocation, ambiguous citations, and broken document hierarchies.
The docling-context sample application (dcuc) addresses these challenges by implementing an offline-first, local-only architecture centered around immutable DocLang (.dclx) packages. By maintaining a unified local context tree, dcuc transforms plain text and unstructured sources into structured, citable nodes. It automates multi-tiered indexing (summary, outline, and full text), provides hybrid retrieval, and exposes both a lightweight CLI and a Gradio-based control plane
TL;DR-What is Docling-Context?
Excerpt from Github repository.
Docling-context is a local context tree that stores documents as immutable DocLang .dclx packages.
See the capability guide, retrieval guide, session guide, memory guide, agent harness guide, CLI reference, and single PDF and folder examples.
Ingest native .dclg, .dclg.xml, and .dclx sources with Ingestor.add_resource. Pass bytes with a filename, or a local Path under an explicit allowed_roots entry. Each revision stores its full document, a short DocLang summary, and a DocLang TOC in one DCLX package. Collection summaries are rebuilt by CollectionWorker; CollectionWorker.status reports whether a summary is stale.
Document conversion is local and optional: install the conversion extra to enable LocalDoclingConverter for Docling's supported formats, including PDF, Office documents, and images. DoclingServeConverter requires an explicit HTTPS endpoint or loopback HTTP endpoint. SQLite tracks logical URIs, revisions, tenant ownership, and an append-only change outbox. Package bytes live under SHA-256 paths. The API is designed so another storage backend can implement the same contracts later.
Docling-context could be used simply by installing the requirements and the implementation is quite easy;
from doclang import DocLangXDocument
from docling_context import LocalContextStore, NodeAddress, Principal
document = DocLangXDocument()
assert document.read_xml('<doclang version="0.7"><text>Hello</text></doclang>')
package = document.write_bytes()
principal = Principal(tenant_id="acme", user_id="alice")
with LocalContextStore("./context-data") as store:
record = store.put(principal, "docling://resources/manuals/hello", package)
citation = NodeAddress(record.document_id, record.revision_id, "/doclang[1]/text[1]")
print(store.read_node(principal, citation).text)
... coupled with native, out-of-the-box ingestion.
from pathlib import Path
from docling_context import CollectionWorker, Ingestor, LocalContextStore, Principal
principal = Principal("acme", "alice")
source_dir = Path("./documents").resolve()
with LocalContextStore("./context-data") as store:
record = Ingestor(store, allowed_roots=(source_dir,)).add_resource(
principal, "docling://resources/manuals/guide", source_dir / "guide.dclx"
)
worker = CollectionWorker(store)
while worker.run_once() is not None:
pass
print(worker.status(principal, "docling://resources/manuals"))
2nd TL;DR-What is Docklang?
Excerpt from Github repository.
DocLang is the AI-native markup format for unstructured content — documents, images, audio, and video alike. It is engineered from the ground up around a single principle: mapping cleanly to LLM tokens. That makes DocLang the most efficient way to read, write, and reason about real-world content with modern AI.
The world's knowledge lives in formats — PDF, HTML, Word, LaTeX, MP3, MP4 — that were designed for rendering or playback, not for understanding. The moment you hand them to an AI system, you lose structure, semantics, layout, timing, or all of them at once. And the formats AI does speak fluently fall short too: Markdown can't describe complex content, HTML wastes tokens, LaTeX is ambiguous, and none of them speak audio or video at all.
Implementation
Flow Architecture
The application's structural design is built upon a strict separation of concerns, ensuring modularity, maintainability, and clean abstraction boundaries throughout the codebase. Rather than allowing client-facing layers to interact directly with underlying storage, parsing, or inference primitives, external entry points—including the command-line interface (cli.py), utility automation scripts, and the interactive web interface built with Gradio (app.py)—delegate all operational workflows to a unified facade (ContextPipeline).
This high-level facade acts as the single point of orchestration, coordinating ingestion, collection indexing, search retrieval, and session logging. Furthermore, because web applications often process user interactions across concurrent worker threads, access to the underlying SQLite database and context store is strictly synchronized using an explicit, reentrant thread lock (RLock), guaranteeing thread-safe execution and data integrity under concurrent loads.
Core Implementation & Code Excerpts
-
Gradio Web User Interface (
app.py): the presentation tier provides interactive tabs for monitoring collection status, running batch ingestions, performing targeted searches, and exploring context trees.
"""Gradio web UI for the docling-context sample.
Run with ``uv run python app.py`` (or ``scripts/start.sh`` for detached mode).
Tabs: Status | Ingest | Search | Ask (chat with citations) | Explore (tree + document).
A single :class:`~dcuc.pipeline.ContextPipeline` is shared by all requests; it
serialises store access internally. SIGTERM/SIGINT close the store cleanly so
``scripts/stop.sh`` performs a graceful shutdown.
"""
from __future__ import annotations
import json
import logging
import signal
import sys
import threading
from pathlib import Path
import gradio as gr
from dcuc.config import Settings
from dcuc.logging_setup import configure_logging
from dcuc.pipeline import TIER_NAMES, ContextPipeline
from dcuc.reporting import answer_markdown, ingest_markdown, write_report
log = logging.getLogger("dcuc.app")
"""
def build_ui(pipeline: ContextPipeline) -> gr.Blocks:
settings = pipeline.settings
# ---- layout ------------------------------------------------------------ #
with gr.Blocks(title="docling-context sample") as ui:
gr.Markdown(
"# docling-context sample\n"
f"Collection `docling://resources/{settings.collection}` - tenant "
f"`{pipeline.principal.tenant_id}` - LLM **{'on' if pipeline.llm else 'off'}** - "
f"vectors **{'on' if pipeline.vectors_enabled else 'off'}**"
)
with gr.Tab("Status"):
status_box = gr.Code(language="json", value=do_status())
gr.Button("Refresh").click(do_status, outputs=status_box)
...
return ui
-
The Application Facade (
pipeline.py): theContextPipelinemanages store interactions, coordinating ingestion, collection jobs, retrieval, and session recording under an explicit threading lock.
"""``ContextPipeline`` - the single facade the CLI and the web UI talk to.
It demonstrates the main docling-context building blocks end to end:
==================== =======================================================
docling-context API Used here for
==================== =======================================================
``Principal`` Tenant/user identity; *every* store call is scoped by it
``LocalContextStore`` SQLite metadata + immutable SHA-256 DCLX packages
``Ingestor`` Convert + summarise + TOC + index one source revision
``CollectionWorker`` Durable jobs that rebuild per-collection summaries
``Retriever`` Scoped lexical / vector / hybrid search with citations
``SessionStore`` Revisioned, append-only conversation transcripts
``NodeAddress`` Stable citation (document, revision, xpath)
==================== =======================================================
Resource management: the store owns an SQLite connection, so the pipeline is a
context manager and must be closed. A single lock serialises store access,
because the web UI runs request handlers on several worker threads while
SQLite connections are not safe for concurrent use.
"""
from __future__ import annotations
import logging
import re
import threading
import time
import uuid
from collections.abc import Callable, Iterator
from dataclasses import dataclass, field
from pathlib import Path
from docling_context import (
CollectionWorker,
ContextResult,
Converter,
Ingestor,
LocalContextStore,
NodeAddress,
PackageError,
Principal,
RecordMissing,
Retriever,
SearchScope,
SessionStore,
)
from .config import Settings
from .converters import MarkdownConverter
from .llm import LLMClient, LLMError
from .summaries import LLMSummaryProvider
from .uris import classify, uri_for, validate_collection
"""
class ContextPipeline:
"""High-level operations over one docling-context store for one principal."""
def __init__(self, settings: Settings, *, llm_client: LLMClient | None = None):
self.settings = settings
self.collection = validate_collection(settings.collection)
# Principal validation (no '/', '%', '..', control chars) happens in the library.
self.principal = Principal(settings.tenant_id, settings.user_id)
self._lock = threading.RLock()
# Opening the store migrates the schema and reconciles packages: any blob
# that is missing or fails its SHA-256 check is marked unavailable.
self.store = LocalContextStore(settings.store_dir)
try:
self.llm = llm_client if llm_client is not None else self._make_llm()
self.retriever = Retriever(self.store, embedder=self._make_embedder())
self.sessions = SessionStore(self.store)
self.worker = CollectionWorker(self.store)
except Exception:
self.store.close() # never leak the SQLite connection on a failed start
raise
-
Canonical URI Generation & Slugs (
uris.py): to prevent ambiguities and ensure safe paths, directory and file names are converted into valid URI segments using custom slugification rules and hash suffixes.
"""Map files in ``input/`` to canonical ``docling://`` URIs.
docling-context rejects ambiguous URIs (encoded separators, ``..``, control
characters, ...). Instead of letting arbitrary filenames reach the store, we
derive a *slug* for each path segment and append a short hash whenever the
slug differs from the original, so two different filenames can never collapse
into the same URI. This mirrors the strategy used by the upstream ``dc`` CLI.
"""
from __future__ import annotations
import hashlib
import os
import re
from pathlib import Path
from docling_context import parse_uri
# Native DocLang sources are always ingestible; others need Docling conversion.
NATIVE_SUFFIXES = (".dclg.xml", ".dclg", ".dclx")
# Converted by the lightweight in-repo converter (no heavy dependencies).
TEXT_SUFFIXES = (".md", ".markdown", ".txt")
# Converted by Docling when the `conversion` extra is installed.
DOCLING_SUFFIXES = (
".pdf", ".docx", ".pptx", ".xlsx", ".html", ".htm",
".png", ".jpg", ".jpeg", ".tif", ".tiff", ".bmp", ".webp", ".csv", ".asciidoc", ".adoc",
)
_COLLECTION_RE = re.compile(r"^[a-z0-9][a-z0-9_-]{0,63}$")
def slug(value: str) -> str:
"""Return a URI-safe segment; append a hash suffix when the value had to change."""
result = re.sub(r"[^\w.-]+", "-", value, flags=re.UNICODE).strip(".-")
if not result:
raise ValueError(f"cannot derive a URI segment from {value!r}")
if result != value:
result += "-" + hashlib.sha256(value.encode("utf-8")).hexdigest()[:8]
return result
def validate_collection(name: str) -> str:
"""Collections are the first URI segment; keep them simple and lowercase.
The ``_context`` name is reserved by docling-context for collection summaries.
"""
if not _COLLECTION_RE.fullmatch(name):
raise ValueError(
"collection must be 1-64 chars of lowercase letters, digits, '-' or '_'"
)
return name
def strip_suffix(name: str) -> str:
"""Remove the (possibly double) extension, e.g. ``a.dclg.xml`` -> ``a``."""
lower = name.lower()
for suffix in (*NATIVE_SUFFIXES, *TEXT_SUFFIXES, *DOCLING_SUFFIXES):
if lower.endswith(suffix) and len(name) > len(suffix):
return name[: -len(suffix)]
return Path(name).stem or name
def classify(path: Path) -> str | None:
"""Return ``native``, ``text``, ``docling`` or ``None`` (unsupported)."""
lower = path.name.lower()
if lower.endswith(NATIVE_SUFFIXES):
return "native"
if lower.endswith(TEXT_SUFFIXES):
return "text"
if lower.endswith(DOCLING_SUFFIXES):
return "docling"
return None
def uri_for(path: Path, base: Path, collection: str) -> str:
"""Build ``docling://resources/<collection>/<sub/dirs>/<stem>`` for ``path``.
Sub-folders of ``input/`` become URI path segments so the context tree mirrors
the folder layout. The result is validated with the library's own parser.
"""
validate_collection(collection)
# Symlinks are deliberately NOT resolved here: the URI follows the name the
# user sees, and the Ingestor's ``allowed_roots`` check (which does resolve
# them) remains the single security boundary for reading files.
relative = Path(os.path.abspath(path)).relative_to(os.path.abspath(base))
segments = [slug(part) for part in relative.parts[:-1]]
segments.append(slug(strip_suffix(relative.parts[-1])))
uri = f"docling://resources/{collection}/{'/'.join(segments)}"
return parse_uri(uri).value # raises InvalidURI if anything slipped through
python
## Supplementary Implementation Modules
To complete the architectural overview of the sample application, three additional core modules manage lightweight format conversion, LLM-backed document summarization, and atomic reporting.
-
Lightweight Markdown Converter (
dcuc/converters.py): while heavy ML-based converters like Docling handle complex PDF and office layouts when opted-in, text-first knowledge bases rely on a streamlined converter. Implemented to satisfy the library'sConverterprotocol,MarkdownConverterparses Markdown and plain-text files directly into structured DocLang XML elements (<heading>and<text>) without heavy dependencies.
class MarkdownConverter:
"""``docling_context.Converter`` implementation for ``.md`` and ``.txt``."""
def __init__(self, *, max_file_bytes: int = 5_000_000):
self.max_file_bytes = max_file_bytes
def fingerprint_options(self) -> dict[str, object]:
# The Ingestor hashes these options together with the source bytes. If the
# options or converter version change, files are re-ingested; otherwise an
# unchanged file is skipped (idempotent ingestion).
return {"converter_version": CONVERTER_VERSION, "max_file_bytes": self.max_file_bytes}
def convert(self, source: bytes, filename: str) -> ConversionResult:
if len(source) > self.max_file_bytes:
raise ValueError(f"{filename}: exceeds {self.max_file_bytes} bytes")
try:
text = source.decode("utf-8-sig")
except UnicodeDecodeError as exc:
raise ValueError(f"{filename}: not valid UTF-8 text") from exc
title = filename.rsplit("/", 1)[-1].rsplit(".", 1)[0]
xml = blocks_to_doclang(parse_markdown(text), title=title)
document = DocLangXDocument()
if not document.read_xml(xml):
raise PackageError(document.last_error() or "generated DocLang is invalid")
return ConversionResult(
package=document.write_bytes(),
converter="dcuc-markdown",
version=CONVERTER_VERSION,
options=self.fingerprint_options(),
)
-
LLM-Backed Summary Provider (
dcuc/summaries.py): the ingestion pipeline can automatically generate concise structural summaries for document revisions using a local LLM. Implementing the library'sSummaryProviderprotocol,LLMSummaryProviderextracts bounded document text, queries the local model for a factual summary, and formats the output into safe DocLang XML. If any error occurs during generation, the provider raises an exception, allowing the ingestor to gracefully fall back to an extractive summary.
class LLMSummaryProvider:
"""Summarise each ingested revision with a local LLM."""
def __init__(self, client: LLMClient):
self.client = client
def summarize(self, document: DocLangXDocument, source_revision: str) -> str:
content = document_text(document)
if not content:
raise ValueError("document has no text to summarise")
summary = self.client.chat(SYSTEM_PROMPT, content, max_tokens=300)
summary = " ".join(summary.split())[:MAX_SUMMARY_CHARS]
log.debug("LLM summary for revision %s: %d chars", source_revision, len(summary))
# The summary sidecar must itself be DocLang; escaping makes it safe.
return f"<doclang><text>{escape(summary)}</text></doclang>"
-
Atomic Timestamped Reporting (
dcuc/reporting.py): every runtime operation—such as ingestion, searching, and question-answering—generates dual machine-readable JSON and human-readable Markdown reports saved to theoutput/directory. To prevent file corruption or half-written artifacts in the event of an interruption,_atomic_writewrites data to a temporary file in the target directory before swapping it atomically usingos.replace.
"""Timestamped reports written to ``output/``.
Every run produces a pair of files that share one timestamp prefix:
``<YYYYmmdd-HHMMSS>_<name>.json`` (machine-readable) and ``.md`` (human-readable).
Writes go to a temporary file first and are renamed, so a crash never leaves a
half-written report behind.
"""
from __future__ import annotations
import json
import os
import tempfile
from dataclasses import asdict, is_dataclass
from datetime import datetime
from pathlib import Path
from typing import Any
from .pipeline import Answer, IngestReport
"""
def _atomic_write(path: Path, content: str) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
fd, tmp = tempfile.mkstemp(dir=path.parent, prefix=".tmp-", suffix=path.suffix)
try:
with os.fdopen(fd, "w", encoding="utf-8") as handle:
handle.write(content)
os.replace(tmp, path)
except BaseException:
Path(tmp).unlink(missing_ok=True)
raise
- Tests and samples: to demonstrate the practical output generated by the application, execution artifacts are automatically serialized into dual formats—a machine-readable JSON log and a structured Markdown report (
output/). For instance, when querying the system regarding the enterprise refund guidelines (What is the refund policy?), the pipeline performs a tiered lexical-vector search across document nodes and constructs a verifiable response complete with precise URI paths, tier levels, XPath selectors, and relevance scoring. As illustrated in the resulting session transcript, if a local LLM invocation is bypassed or unavailable, the framework seamlessly switches to an extractive fallback method, quoting the exact passages verbatim while preserving exact provenance metadata.
{
"question": "What is the refund policy?",
"answer": "Most relevant passages:\n[1] Refund Policy Customers may request a full refund within 30 days of purchase for annual plans. Monthly plans Monthly plans are not refundable, but cancellation stops the next renewal. How to request a refund Open a ticket with the billing team and include the invoice number. Refunds are issued to the original payment method within 5 business days.\n[2] Refund Policy\n[3] Refund Policy",
"method": "extractive",
"citations": [
{
"number": 1,
"uri": "docling://resources/knowledge/policies/refund-policy",
"xpath": "@summary:/doclang[1]/text[1]",
"tier": 0,
"score": 1.06,
"text": "Refund Policy Customers may request a full refund within 30 days of purchase for annual plans. Monthly plans Monthly plans are not refundable, but cancellation stops the next renewal. How to request a refund Open a ticket with the billing team and include the invoice number. Refunds are issued to the original payment method within 5 business days.",
"document_id": "a67aee64fbb8429baa9afe36f9448c00",
"revision_id": "448341261cf843b39dc5dd17cae59f69"
},
{
"number": 2,
"uri": "docling://resources/knowledge/policies/refund-policy",
"xpath": "/doclang[1]/heading[1]",
"tier": 1,
"score": 0.53,
"text": "Refund Policy",
"document_id": "a67aee64fbb8429baa9afe36f9448c00",
"revision_id": "448341261cf843b39dc5dd17cae59f69"
},
{
"number": 3,
"uri": "docling://resources/knowledge/policies/refund-policy",
"xpath": "@toc:/doclang[1]/toc[1]/entry[1]/description[1]",
"tier": 1,
"score": 0.53,
"text": "Refund Policy",
"document_id": "a67aee64fbb8429baa9afe36f9448c00",
"revision_id": "448341261cf843b39dc5dd17cae59f69"
}
],
"session_id": "ade35b2fa47b433495cc9ef5ce3f9fa4"
}
Conclusion
By seamlessly fusing lightweight local SQLite metadata indexing with immutable, content-addressed DocLang .dclx package storage, docling-context empowers developers and architects to construct highly resilient, offline-first document intelligence systems. This robust architectural foundation eliminates external cloud dependencies while guaranteeing exact, stable node citations across document revisions. Furthermore, by incorporating multi-tiered search strategies—such as prioritized body-first retrieval alongside robust fallback mechanisms for local LLM inference—the framework ensures continuous, dependable operational performance even when auxiliary services encounter network disruptions or runtime failures.
Thanks for reading 📃
Links
- Code repository for this post: https://github.com/aairom/docling-context-uc
- Docling.ai: https://docling.ai/
- Doclang: https://github.com/doclang-project
- Docling-Context repository: https://github.com/docling-project/docling-context
Top comments (0)