On-premise structured extraction with LLM using Ollama

This post is also on cocoindex.io, with interactive figures. The code uses CocoIndex v1.

A PDF manual holds structured facts: a module name, its classes, its functions, and the arguments each function takes. The layout makes those facts easy for a person to read and hard for a program to use. You cannot filter a folder of PDFs for "functions that take a memo argument", join it against your own tables, or count what a new version added.

This post builds a pipeline that turns each PDF into a typed Postgres row on your own hardware. docling converts the PDF to Markdown. A local Ollama model fills a nested Pydantic schema. CocoIndex writes one row per document and keeps the table in sync with the folder. With LLM_MODEL set to an ollama/ model, the documents stay on your machine. After the first build, only new or edited PDFs go through the model.

The full example is in the CocoIndex repo under examples/manuals_llm_extraction, and there is a step-by-step docs walkthrough.

The short version

• Output: one Postgres row per PDF, with summary columns plus the full nested ModuleInfo as JSON.
• Data boundary: with an ollama/ model, docling, Ollama, and Postgres run on hosts you control, so document text does not go to a hosted API.
• Schema as instruction: a nested Pydantic model tells the LLM what to extract at each level (module, class, method, argument).
• Incremental: @coco.fn(memo=True) skips unchanged files, so a re-run spends LLM calls only on files that changed.
• Model choice: one environment variable, LLM_MODEL, selects ollama/llama3.2, another local model, or a hosted one.

Why extract fields, not only embed chunks

The usual first step with a pile of documents is to split them into chunks, embed the chunks, and search by similarity. That works for "find passages about X". It does not work for questions that need every match, an exact count, or a filter on a field. Those questions need the facts as typed columns.

Use both: vector search for open questions in natural language, and extracted rows for filters, counts, and joins that need an exact answer. The PDF elements post covers the embedding side.

Why run the model on your own hardware

Extraction sends the whole document to the model, not a short query. With a hosted API, every page of every contract, filing, or internal manual leaves your network. Teams with confidentiality rules often cannot send these documents to a hosted API at all.

Running the model with Ollama changes three things:

• Data boundary. Document text goes from disk to docling to Ollama to Postgres, all on hosts you control.
• Cost model. You pay for hardware, not per token. Re-extracting a folder after a schema change costs GPU time, not an API bill.
• Throughput. Your GPU sets the limit, not a provider's rate limit.

The trade-off is extraction quality. llama3.2 is a 3B-parameter model that runs on a laptop, and it is likely to miss more on long, dense documents than a large hosted model. Check its output on a sample of your own PDFs. If it falls short, pull a larger local model and change one variable (see Swap the model, keep the pipeline).

Where the same pipeline fits

The example extracts Python module docs because the sample PDFs are public. The pattern applies to any document set whose facts follow a shape. To adapt it, you change the Pydantic model and the prompt. The folder source, docling, Ollama, memoization, and the Postgres target stay as they are. Pass the prompt to @coco.fn(memo=True, deps=EXTRACT_PROMPT) so that editing it re-runs the extraction; CocoIndex tracks a function's code, not module-level constants it reads (deps).

CocoIndex re-runs only what changed

CocoIndex is an incremental data transformation framework. You declare target_state = transformation(source_state) in plain async Python with your own types. A Rust engine underneath tracks changes, processes only what changed, and manages the target tables. When you edit one manual, CocoIndex re-parses and re-extracts that one and updates its row in place. The example is one main.py of 187 lines.

Ollama serves the model locally

Ollama runs open LLMs on your machine. Install it, then pull a model:

ollama pull llama3.2

CocoIndex reaches Ollama through LiteLLM, so the model is a string: ollama/llama3.2. Change it to ollama/mistral, ollama/qwen2.5, or a hosted provider such as openai/gpt-4o, and the rest of the pipeline stays the same. For a fully on-premise setup, keep it on an ollama/ model.

Overview of the pipeline

The flow watches a manuals/ folder and turns each PDF into one typed Postgres row:

1. Read: CocoIndex walks manuals/ and reads new or updated PDF files.
2. Convert: docling parses the document into Markdown on a coco.GPU runner, so the parse does not block the event loop.
3. Extract: the Markdown goes to Ollama through instructor, which validates the output against a typed ModuleInfo (title, description, classes, methods, arguments).
4. Declare: one Postgres row per manual holds summary counts plus the full structure as JSON.
5. Skip: on later runs, an unchanged file skips both the parse and the extraction.

Define the pipeline

The entry point is an app_main function wrapped in a coco.App. It mounts one Postgres table target, walks the manuals/ folder for PDFs in live mode, and runs one processor per file. The whole example is in one main.py.

```python title=main.py
import cocoindex as coco
from cocoindex.connectors import localfs, postgres
from cocoindex.resources.file import FileLike, PatternFilePathMatcher

@coco.fn
async def app_main(sourcedir: pathlib.Path) -> None:
table = await postgres.mount_table_target(
PG_DB,
table_name=TABLE_NAME,
table_schema=await postgres.TableSchema.from_class(
ModuleRecord, primary_key=["filename"]
),
pg_schema_name=PG_SCHEMA_NAME,
)

files = localfs.walk_dir(
    sourcedir,
    recursive=True,
    path_matcher=PatternFilePathMatcher(included_patterns=["**/*.pdf"]),
    live=True,
)
await coco.mount_each(process_file, files.items(), table)

app = coco.App(
coco.AppConfig(name="ManualsLlmExtraction"),
app_main,
sourcedir=pathlib.Path("./manuals"),
)

With `live=True` and `cocoindex update -L`, the source keeps watching the folder, so a manual added later is picked up without a restart. A plain `cocoindex update` scans once and exits. `coco.mount_each` runs `process_file` once per matched PDF.

### The extraction schema

The output type is nested Pydantic. instructor sends the schema with the request, so the field names and nesting tell the model what to extract. A `ModuleInfo` has `classes` (each with `methods`) and module-level `methods` (each with `args`). The model sees the expected fields at every level, so you do not write a separate prompt for each level.



```python
class ArgInfo(pydantic.BaseModel):
    name: str
    description: str = ""

class MethodInfo(pydantic.BaseModel):
    name: str
    args: list[ArgInfo] = pydantic.Field(default_factory=list)
    description: str = ""

class ClassInfo(pydantic.BaseModel):
    name: str
    description: str = ""
    methods: list[MethodInfo] = pydantic.Field(default_factory=list)

class ModuleInfo(pydantic.BaseModel):
    title: str
    description: str
    classes: list[ClassInfo] = pydantic.Field(default_factory=list)
    methods: list[MethodInfo] = pydantic.Field(default_factory=list)

Convert PDF to Markdown

Ollama does not read PDFs, so each document is converted to Markdown first. docling does the parse. The function is decorated @coco.fn.as_async(runner=coco.GPU), so it runs on the GPU runner instead of blocking the event loop. The example configures docling for CPU; change AcceleratorDevice to use a GPU.

@functools.cache
def pdf_converter() -> DocumentConverter:
    options = PdfPipelineOptions(
        accelerator_options=AcceleratorOptions(device=AcceleratorDevice.CPU)
    )
    return DocumentConverter(
        format_options={InputFormat.PDF: PdfFormatOption(pipeline_options=options)}
    )

@coco.fn.as_async(runner=coco.GPU)
def pdf_to_markdown(content: bytes) -> str:
    source = DocumentStream(name="manual.pdf", stream=io.BytesIO(content))
    return pdf_converter().convert(source).document.export_to_markdown()

Extract structured data with Ollama

The Markdown goes to the local model. instructor wraps LiteLLM and validates the response against ModuleInfo, so you get a typed object back instead of free text to parse. The model name comes from context (LLM_MODEL); set it to ollama/llama3.2 for an on-premise run.

@coco.fn(memo=True)
async def extract_module(markdown: str) -> ModuleInfo:
    client = instructor.from_litellm(litellm.acompletion, mode=instructor.Mode.JSON)
    result = await client.chat.completions.create(
        model=coco.use_context(LLM_MODEL),
        response_model=ModuleInfo,
        messages=[
            {"role": "system", "content": EXTRACT_PROMPT},
            {"role": "user", "content": markdown},
        ],
    )
    return ModuleInfo.model_validate(result.model_dump())

@coco.fn(memo=True) caches the result by its input Markdown. While the Markdown, the function's code, and LLM_MODEL are unchanged, later runs do not send it to the model again.

Declare the Postgres row

process_file ties the steps together: read the file, convert it, extract, then declare one row. You describe the row you want, and CocoIndex inserts, updates, or deletes rows to match. The row carries the summary counts and the full ModuleInfo as JSON. The primary key is the file name without its folder, so two x.pdf files in different subfolders would share a row; use the relative path if your folder has that case.

@dataclass
class ModuleRecord:
    filename: str  # primary key
    title: str
    description: str
    num_classes: int
    num_methods: int
    module_info: str  # the full ModuleInfo as JSON

@coco.fn(memo=True)
async def process_file(
    file: FileLike,
    table: postgres.TableTarget[ModuleRecord],
) -> None:
    markdown = await pdf_to_markdown(await file.read())
    info = await extract_module(markdown)
    table.declare_row(
        row=ModuleRecord(
            filename=file.file_path.path.name,
            title=info.title,
            description=info.description,
            num_classes=len(info.classes),
            num_methods=len(info.methods),
            module_info=json.dumps(info.model_dump()),
        )
    )

Providing shared resources

The Postgres pool and the model name are provided once at startup with @coco.lifespan and read inside the functions with coco.use_context. LLM_MODEL is declared with detect_change=True, so when you change models, CocoIndex re-runs process_file for every file against the new one, with no cache to clear.

PG_DB = coco.ContextKey[asyncpg.Pool]("manuals_db")
LLM_MODEL = coco.ContextKey[str]("llm_model", detect_change=True)

@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
    async with asyncpg.create_pool(os.environ["POSTGRES_URL"]) as pool:
        builder.provide(PG_DB, pool)
        builder.provide(LLM_MODEL, os.environ.get("LLM_MODEL", "openai/gpt-4o"))
        yield

Run it on the sample manuals

Prerequisites

• Python 3.11+
• Ollama installed, with a model pulled: ollama pull llama3.2
• Postgres for the output table

Two settings matter for a fully local run. docling downloads its layout models on first use, so an air-gapped host needs them cached in advance. Ollama's default context window can be shorter than a long manual; if extractions look cut off, raise num_ctx for the model.

Quick start

The example ships a manuals/ folder with 3 Python module reference PDFs (array, base64, copy).

Set LLM_MODEL before the first run. The example falls back to openai/gpt-4o when LLM_MODEL is not set, and .env.example ships with that value. For an on-premise run, set LLM_MODEL=ollama/llama3.2 in .env first, or the documents go to OpenAI.

cd examples/manuals_llm_extraction

# Start Postgres
docker compose -f ../../dev/postgres.yaml up -d

# Configure and install
cp .env.example .env      # set POSTGRES_URL and LLM_MODEL=ollama/llama3.2
pip install -e .

# Build the index
cocoindex update main     # catch-up run
cocoindex update -L main  # or: live run, keep watching the manuals/ folder

Query the results

Each manual becomes one row in coco_examples.modules_info. The counts follow each module's shape: in the docs walkthrough run, base64 comes out as 22 module-level functions and no classes, and array as a single class. That run does not name its model; expect a 3B local model to differ, and compare on your own documents.

SELECT filename, title, num_classes, num_methods FROM coco_examples.modules_info;

-- pull the full nested structure for one module
SELECT module_info::jsonb -> 'classes' -> 0 -> 'methods'
FROM coco_examples.modules_info WHERE filename = 'copy.pdf';

-- every module-level function that takes a `memo` argument
SELECT filename, m ->> 'name' AS function
FROM coco_examples.modules_info,
     jsonb_array_elements(module_info::jsonb -> 'methods') AS m
WHERE m -> 'args' @> '[{"name": "memo"}]';

Incremental updates in action

After the first build, the parse and LLM work in a run depend on what changed in the folder, not on its size. (Each run still lists the folder and checks each file's modification time.) An unchanged PDF skips both the docling parse and the LLM call. A changed PDF is re-parsed and re-extracted, and its row is updated in place. A new PDF gets its own row, and a deleted PDF's row is dropped.

Try this sequence, one run each: the first build, a run with no changes, then edit copy.pdf, add struct.pdf, delete base64.pdf, and switch the model. It makes 8 LLM calls, against 19 for a script that re-extracts the whole folder on every run. The gap grows with the folder: with 10,000 PDFs and 50 edits per day, a daily run makes 50 LLM calls instead of 10,000. On a local model, the skipped calls leave the GPU free for other work.

Re-run cocoindex update main at any time, or leave -L running to watch the folder.

Swap the model, keep the pipeline

The model is one context value, so moving from local to hosted, or between local models, is one environment variable:

LLM_MODEL=ollama/llama3.2          # fully on-premise
LLM_MODEL=ollama/qwen2.5           # a different local model
LLM_MODEL=openai/gpt-4o            # hosted, when privacy isn't a constraint

Because LLM_MODEL is declared with detect_change=True, a change re-runs process_file for every file (the docling parse and the extraction) against the new model.

Community

Find us on GitHub and Discord. If you adapt the pipeline to your own documents, we would like to hear what schema you used.