Canonical version: https://thelooplet.com/posts/ontology-guided-extraction-vs-extractbench-cutting-duplication

Ontology-Guided Extraction vs ExtractBench: Cutting Duplication

TL;DR: Ontology‑guided extraction slashes duplicate entities by 94 % compared with schema‑guided pipelines, while still delivering higher recall and lower cost on enterprise document streams.

Introduction

Enterprise knowledge graphs crumble when the same entity appears under multiple aliases or when relationship tuples are emitted twice. A recent production system that injects a live slice of a curated ontology into the prompt of a Qwen3.5‑9B model reduced the catalog‑size overhead by ~94 % and lifted search recall from ~70 % to 95 % on intelligence corpora (arXiv:2607.28662). At the same time, the ExtractBench benchmark revealed that schema‑guided agents still struggle with long record lists, truncating up to 30 % of rows on multi‑page invoices (arXiv:2607.29677). The tension is clear: a model‑centric, ontology‑aware pipeline can prune duplication aggressively, while schema‑driven agents excel at strict adherence to user‑defined field layouts but pay a steep cost in processing time and missed entities.

The thesis of this piece is that developers building enterprise extraction stacks should pivot to ontology‑guided prompting for deduplication‑heavy workloads and reserve schema‑guided agents for compliance‑critical forms where exact field placement trumps recall. The following sections break down the two approaches, show how to wire them into a Kafka‑driven ingestion pipeline, and compare their operational footprints on real‑world data.

Ontology‑Guided Extraction

Live Ontology Slice Retrieval

The core innovation is a dynamic ontology slice fetched at extraction time. Instead of loading a monolithic taxonomy (often dozens of MB), the system queries a Neo4j graph for the top‑k concepts whose embeddings are closest to the document’s topic vector. This similarity search runs in ≈12 ms per document on a single RTX 4090, shaving the prompt token budget from 2 400 to 150 tokens. The slice is then interpolated into the prompt:

import torch, json, requests
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')

def fetch_ontology_slice(topic_vec, k=25):
    resp = requests.post(
        'https://kg.example.com/ontology/slice',
        json={'vector': topic_vec.tolist(), 'k': k}
    )
    return resp.json()['concepts']  # list of {id, label, definition}

def build_prompt(doc_text, ontology_slice):
    slice_text = "\n".join([f"- {c['label']}: {c['definition']}" for c in ontology_slice])
    return (
        f"You are extracting entities aligned to the following ontology:\n{slice_text}\n\n"
        f"Document:\n{doc_text}\n\n"
        "Return JSON with entity IDs, canonical names, and source spans."
    )

The prompt injection forces the model to anchor every extracted label to a known concept, preventing drift into ad‑hoc taxonomy creation that fuels duplication.

Two‑Pass Extraction Flow

The pipeline runs two inference passes per document chunk:

1. Pass 1 extracts raw entities and their surface forms.
2. Deterministic Cleaning normalizes case, strips honorifics, and canonicalizes dates using dateutil.
3. Chunk‑Level Merging de‑duplicates within the same PDF page using a hash of the normalized name.
4. Pass 2 re‑runs the model with a relationship‑focused prompt that only references entities survived after step 3.
5. Six Deduplication Algorithms (string‑match, Levenshtein, Jaro‑Winkler, token‑set ratio, embedding‑cosine, and rule‑based alias tables) run without any additional LLM calls, guaranteeing deterministic latency.

Conflict Guard via Embedding Resolution

When two candidate entities survive all deterministic filters, the system computes a conflict‑resolution embedding. The higher‑scoring embedding must exceed a 0.82 cosine similarity threshold; otherwise the merge is rejected, and both entities are retained with a “possible duplicate” flag. This guard eliminates silent conflation—an issue that caused 7 distinct bugs in the baseline system, including a single‑character truncation that broke downstream joins.

Production Metrics

• Throughput: 1 200 docs /min on a 3‑node Kafka consumer cluster (each node 8 vCPU, 32 GB RAM).
• Latency: 210 ms average per document (including Neo4j slice fetch).
• Recall: 95 % (vs 70 % baseline) on the internal intelligence corpus.
• False Merges: 0 % across 5 M extracted entities.
• Cost: $0.12 per 1 000 pages on a spot‑instance GPU fleet, ≈78 % cheaper than a comparable schema‑guided VLM deployment (see ExtractBench results).

Schema‑Guided Extraction (ExtractBench)

Benchmark Overview

ExtractBench aggregates 4 869 pages across 370 enterprise documents in eight business domains, measuring three orthogonal metrics: value accuracy (order‑insensitive F1), grounding (word‑ and page‑level F1), and cost (GPU‑hours per 1 000 pages). The benchmark stresses record completeness by embedding long itemized tables (e.g., purchase orders with >200 rows).

Agent Landscape

Agent	Value F1	Word‑Ground F1	Page‑Ground F1	Cost (GPU‑h/1k pg)
LlamaExtract Agentic Plus	0.92	0.94	0.93	0.45
Commercial VLM (GPT‑4‑V)	0.78	0.81	0.79	1.12
Coding Agent (Python‑based)	0.88	0.85	0.84	2.03

LlamaExtract tops all three metrics while spending ≈60 % less GPU time than the leading commercial VLM. However, all agents exhibit a 30 % truncation rate on record lists longer than 100 rows, a direct consequence of context‑window limits.

Prompt Engineering vs. Schema Enforcement

ExtractBench agents receive a JSON schema that enumerates expected fields and optional repeatable sections. The prompt instructs the model to “return a JSON object matching the schema and include a source_spans array for each field.” This schema‑first approach guarantees structural compliance but does not prevent the model from inventing duplicate entities when the same person appears under different aliases across pages.

Cost‑Accuracy Trade‑off

The benchmark’s cost model reveals a diminishing‑returns curve: doubling GPU allocation from 0.5 h to 1 h per 1 000 pages improves value F1 by only +0.04 points, while the cost skyrockets. Teams that prioritize budget over perfect completeness should cap inference at the 0.5 h tier and accept the truncation penalty.

Implementing a Hybrid Pipeline

Architectural Sketch

+-------------------+      +-------------------+      +-------------------+
|   Kafka Topic     | ---> |   Consumer (Py)  | ---> |   Ontology Slice   |
| (doc metadata)   |      |   (asyncio)       |      |   Service (REST)   |
|                     |                     |
v                     v                     v
+-----------+           +-----------+           +-----------------+
| PDF/Img   |           | Qwen Model|           | Deduplication   |
| Handler   |           | Inference |           | Engine (det.)   |
+------------->+-------------------------------+<------------+
|   KG Ingest (Neo4j)            |
+-------------------------------+

Code Walkthrough (Python 3.11, AsyncIO)

import asyncio, json, aiohttp
from confluent_kafka import Consumer
from transformers import AutoModelForCausalLM, AutoTokenizer

KAFKA_CONF = {
    'bootstrap.servers': 'kafka:9092',
    'group.id': 'kg-extractor'
}
consumer = Consumer(KAFKA_CONF)
consumer.subscribe(['doc-stream'])

model = AutoModelForCausalLM.from_pretrained(
    'Qwen/Qwen3.5-9B', device_map='auto')
tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen3.5-9B')

async def fetch_slice(topic_vec):
    async with aiohttp.ClientSession() as s:
        async with s.post(
            'https://kg.example.com/ontology/slice',
            json={'vector': topic_vec.tolist(), 'k': 25}
        ) as r:
            return await r.json()

async def process_message(msg):
    payload = json.loads(msg.value())
    doc_text = extract_text(payload['uri'])   # PDF/Office handler omitted
    topic_vec = embed_topic(doc_text)         # e.g., SentenceTransformer
    slice_cfg = await fetch_slice(topic_vec)
    prompt = build_prompt(doc_text, slice_cfg['concepts'])
    inputs = tokenizer(prompt, return_tensors='pt').to('cuda')
    out = model.generate(**inputs, max_new_tokens=1024)
    raw = tokenizer.decode(out[0], skip_special_tokens=True)
    entities = json.loads(raw)['entities']
    cleaned = deterministic_clean(entities)
    merged = chunk_merge(cleaned)
    rel_prompt = build_relation_prompt(merged, slice_cfg['concepts'])
    # ...repeat inference
    final = dedup_engine(merged)   # six-algorithm cascade
    await ingest_to_kg(final)

async def main():
    while True:
        msg = consumer.poll(1.0)
        if msg is None:
            continue
        asyncio.create_task(process_message(msg))

asyncio.run(main())

The deterministic_clean, chunk_merge, and dedup_engine functions implement the five‑stage refinement pipeline described in the paper. They rely only on Python stdlib and rapidfuzz for fuzzy matching, guaranteeing zero‑LLM latency for those steps.

Adding a Schema‑Guided Fallback

if payload['type'] in COMPLIANCE_FORMS:
    await run_schema_agent(payload['uri'], payload['schema_id'])
else:
    await process_message(msg)

The run_schema_agent wrapper loads the user‑provided JSON schema, builds a schema‑first prompt, and invokes a VLM (e.g., LlamaExtract). The result is merged back into the KG with a provenance flag indicating “schema‑origin”. This hybrid approach preserves high recall for free‑form intel documents while ensuring regulatory fidelity for mandated forms.

Evaluation & Benchmarks

Duplicate‑Entity Reduction

On a test set of 2 M extracted entities from mixed‑format PDFs, the ontology‑guided pipeline reported 94 % fewer duplicate clusters than the baseline schema‑guided run (average cluster size 1.04 vs 1.68). The six deterministic deduplication stages contributed ≈60 % of the reduction; the live ontology slice contributed the remaining ≈34 % by preventing alias creation at source.

Recall vs Cost Curve

GPU‑hours/1k pg	Value F1 (Schema)	Recall (Ontology)
0.5	0.78	0.91
1.0	0.82 (+0.04)	0.94 (+0.03)
2.0	0.84 (+0.02)	0.95 (+0.01)

The ontology‑guided system reaches 95 % recall at 0.45 GPU‑h/1k pg, beating the schema‑guided VLM that needs 1.12 GPU‑h to hit 0.78 F1. The marginal returns flatten beyond 1 GPU‑h for both, confirming the cost‑efficiency sweet spot lies near 0.5 GPU‑h.

Grounding Accuracy

Grounding metrics (word‑level F1) were 0.94 for the ontology pipeline versus 0.81 for the schema pipeline. The difference stems from the ontology’s entity‑centric prompting, which forces the model to emit source spans for each canonical entity, whereas schema agents sometimes collapse adjacent fields into a single span to satisfy JSON shape constraints.

Failure Modes

Symptom	Root Cause	Mitigation
Truncated record list	Context‑window overflow	Chunk the table into overlapping tiles (as in ExtractBench)
Silent conflation of “John A. Doe” and “J. Doe”	Missing alias table	Populate alias KB from HR master data
High false‑pass rate in rule engine	Over‑broad regexes	Introduce rule‑aware evidence routing (region vs page)

The evidence‑grounded constraint checking work from construction documents (arXiv:2607.29058) informs the mitigation: allocating one overview tile and three overlapping tiles improved decision accuracy by 10.6 pp. Applying the same tiling to long tables recovers up to 12 % of lost rows.

What This Actually Means

The real story is not that LLMs magically replace traditional rule‑based pipelines; it is that ontology‑guided prompting transforms the LLM into a deterministic deduplication engine. By anchoring every extraction to a curated concept, you eliminate the most costly post‑processing step—entity resolution. Teams that adopt this pattern now will enjoy sub‑second latency, near‑zero false merges, and budgetary headroom to scale to millions of documents per day. The mistake most architects make is to default to a pure schema‑first VLM because it “covers all fields”. In practice, that choice inflates GPU spend by ≥2× and still leaves you with a noisy KG that must be cleaned downstream. The smarter path is a dual‑track architecture: ontology‑guided for free‑form intel, schema‑guided only where regulatory compliance mandates exact field placement.

Key Takeaways

• Deploy a live ontology slice via embedding similarity; it slashes prompt size and cuts duplicate‑entity clusters by ~94 %.
• Run two inference passes: first for entities, second for relationships, to keep the model focused and reduce hallucinations.
• Implement six deterministic deduplication algorithms (string, Levenshtein, Jaro‑Winkler, token‑set, embedding‑cosine, alias rules) outside the LLM to guarantee O(1) latency.
• Use ExtractBench as a benchmark to quantify value‑accuracy, grounding, and cost; aim for ≤0.5 GPU‑h/1k pg to hit the sweet spot.
• Reserve schema‑guided agents for compliance‑critical forms; otherwise, the ontology‑centric pipeline delivers higher recall at a fraction of the price.

References

• An Ontology‑Guided, Deduplication‑Aware Extraction Layer for Knowledge Graph Construction from Heterogeneous Documents (https://arxiv.org/abs/2607.28662) — arXiv CS.AI
• Evidence‑Grounded Constraint Checking in Construction Documents (https://arxiv.org/abs/2607.29058) — arXiv CS.AI
• ExtractBench: A Benchmark for Schema‑Guided Enterprise Document Extraction (https://arxiv.org/abs/2607.29677) — arXiv CS.AI

See more articles on The Looplet

Further reading

Read Next

• How to Build SelfImproving LLM Agents with Recursive Harness Loops
• Best Way to Safely Deploy AIGenerated Satellite Images in GeoApplications
• EvoPINN vs Handtuned PINNs: Automated Design Wins

Read next: continue with one of these related guides.

---

Originally published at The Looplet.